Gigabit File uploads Over HTTP
Please see the NODE.js version of this article.
Updated version that uses NGINX reverse proxy and asynchronous code
Large file uploads how do I love thee? Let me count the ways. These days having to deal with large file uploads is rather common place in IT environments, and by large file uploads I mean files that are over 500 MB in size. Sure we have replication technologies that can help us to keep our data in sync, but there is still that need to move large amounts of data on a regular basis.
Most of the times
that I had to move anything over 500 MB in size I would typically split the file
into several smaller files using a file compression utility and then upload these
smaller files via ftp or secure shell (ssh). Once all of the smaller files have been
uploaded I would use the file compression to recreate the original file. However
that required the setup of an FTP server or a Secure Shell server and the use
of a third party file compression utility.
So I asked "With the prevalence of web browsers and web servers in the IT
environment could I accomplish the same
thing using the HTTP protocol?"
A quick search on the Internet showed that web servers will
typically limit the maximum file size that can be uploaded to anywhere between
2 GB and 4 GB, and in addition to that most web browsers will allow you to upload
around 2 GB. I suppose the reason for that is that the Content-Length header is being
treated as a signed 32-bit integer in the web browser and the maximum size of a
signed 32-bit integer is 2,147,483,647.
As I continued to search I began to look at the HTML 5
specification and the APIs that are a
part of that specification because the indication was that these new APIs would
allow me to upload files greater than 2 GB in size over the HTTP protocol. I
also came across examples of code on the Internet that indicated what could be
done, but not a complete example of how it could be done.
