[ReactFilePondUpload] Uploading huge files
Forge component by Stuart Harris
Application Type
Reactive

We have a scenario where we upload a couple hundreds MB files, but our server responses with 413 - payload too large. It happens that the component uploads the whole file in a single request and the server doesn't like that. 

Currently we don't have an option to change the server config to accept bigger requests.

Would be possible to add a feature that if a file is bigger than a certain (customizable) threshold it internaly splits the upload to several http requests?

Solution

Hi Adam,

Thank you for your message, and for explaining clearly your situation.

I imagine the recent update could resolve the problem for you as it splits files into chunks.

However, I am making an assumption that you are not using the latest version.

Please upgrade to the latest version, and it would be great if you would let me know how it goes.

If you are already on the latest version, let me know too, and I will investigate.

Kind regards,

Stuart.

Hi Stuart,

you were right, the latest component version did resolve the issue, thank you!

One more note though, I've noticed a significant performance drop compared to a single request upload (400MB file on 100Mbit upload speed is supposed to upload in about 45 seconds) while on chunks upload it took over 9 minutes (which gives an averege 5.8Mbit upload speed), which I guess is caused by huge database overhead (loading file part from database, appending a chunk and saving to database) as the upload slowed down the more chunks I've uploaded (check attached image of upload spikes and gaps with longer gaps the more of the file was uploaded).

Is the database overhead necessary? And is it possible to change the default chunks size (from about 30MB) to a custom value (90MB would perform the best in our case) so there would be less processing?

Thank you. Adam 

uploadspikes.png

Hi Adam,

Thank you for your message.

It is not currently possible to modify the chunk size.  It is a balance between the ability to handle large files, performance and keeping the timeouts low.

You are correct about the cause of the delay.

I surprised by the factor of difference from 45 seconds to 9 minutes, in my testing it was slower but not to that extent.  It might be worth testing the upload time using a different component that uploads the whole file is a single HTTP request; unless you have already done so, that is.

I am working on improving the performance by threading the chunked uploads, so an upcoming release will improve performance.  So that will appear in an upcoming release

I hope this helps.

Kind regards,

Stuart

Hi Stuart,

thank you for your response. I guess that the performance may vary due to a server performence, we run Outsystems on premise with a highly powerful server which is also under high workload. It is great to hear that you are aware of the issue.

Yes, I have tested a single thread upload, but to be fair the file was just shy of 100MB (due to a single request size limitation) and I've calculated the results accordingly.

Thank you for listening to a feedback, it's truly appreciated. Also great work with the extension, it is absolutely awesome!

With best regards while looking forward to future releases, Adam

Hi Adam,

Thank you for your kind words!

Cheers,

Stuart

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.