App Engine with Ruby, Python, and Perl Reply hooks in Gmail; A case study in over-engineering
Apr 10

Gears Future APIs: Resumable Uploads via PUT/POST

Gears, Tech Add comments

Resume Upload

I have sat watching an upload happening, with a spinning “I am doing something” indicator, wondering how long to go, and if anything is actually really happening. Some sites do a better job and give you some feedback, but most still don’t.

If the Internet connection goes down and you are at 90% you cry out as you know that you will have to tell the browser to send it all up again.

This is why we want to implement Content-Range for HTTP POST and PUT:

The primary primitive necessary to enable large file uploads is the ability to specify byte ranges in POST and PUT requests. Byte ranges are already standardized for GET requests, and there are implementations of byte-range PUT used by WebDAV, but to our knowledge there has been little effort to use them for POST. Byte-range POST/PUT could be used to resume incomplete transfers, or to explicitly break transfers into smaller sized chunks. We propose standardizing byte range POST/PUT in a manner analogous to byte range GET.

The use of Content-Range headers in POST/PUT is allowed by the standard. Section 9.6 explicitly suggests the possibility of using Content-Range headers in PUTs: “The recipient of the entity MUST NOT ignore any Content-* (e.g. Content-Range) headers that it does not understand or implement and MUST return a 501 (Not Implemented) response in such cases.” However, such functionality has not been widely deployed, and therefore there exists no reference implementation or standardized semantics for how it should be used.

Use Cases

The use of Content-Range POST/PUT could be used to compose a chunked transfer protocol where a large POST/PUT was broken up into many small POST/PUTs. This would be similar to the protocols used by several current resumable uploaders. The basic algorithm for this model is to continue sending each chunk (probably in serial order) until it is successfully acknowledged. Note that if used for POSTs, some server-specific mechanism would be required to uniquely identify the the set of chunks constituting a single logical transfer. If it is necessary to accommodate server-side chunk loss (e.g. due to failures partially masked by replication) some additional custom protocol components would be necessary for the server to indicate to the client which chunks it had successfully received.

For some web applications, it may be desirable to upload a subset of a file to the server to reduce the latency between when a file is selected, and when the user can manipulate it in the application. A concrete example of this might be an image hosting website, which would like to upload the 64KB EXIF data segment from a JPEG file, so that the user can quickly view a thumbnail version of the file in the application while the rest of the image is transferred in the background.

It makes sense for this to work with the Blob API, adding:

[java]
interface Blob {

Blob slice(int64 offset);
Blob slice(int64 offset, int64 length);

}
[/java]

Other Future APIs

Disclaimer: This is me rambling about APIs and tools that I would love to see in Gears, or the Open Web as a whole. Do you have ideas for cool Gears that make the Web better? Let us know!.

One Response to “Gears Future APIs: Resumable Uploads via PUT/POST”

  1. C. Brown Says:

    I don’t know a thing about technology, but boy would I love to have the feature you describe in my browser. Especially that one where I can upload thumbnails first and the remainder of the picture files later.

    GO GEARS!

Leave a Reply

Spam is a pain, I am sorry to have to do this to you, but can you answer the question below?

Q: What are the first four letters in the word British?