[ReactFilePondUpload] Uploading numerous images causes filepond upload to be droppping performance

Forge Component
(5)
Published on 27 May by Stuart Harris
5 votes
Published on 27 May by Stuart Harris

Hi, I have tested uploading 1500 images using this component and observed that it is dropping performance compared when I was uploading only 300 images. Anyone also experiencing this kind of scenario? Also I could present benchmarking times with 300 images vs 1500 images


When Uploading 300/300 images, it only took 7 minutes to finish the whole process from reading all the files to uploading it to the server

When Uploading 1500 images, 300/1500 took 30 minutes to finish the whole process with same above.

HI Jaybriel,

Thank you for performing this test and posting the results.

The network tab in your browsers dev tools should give you a good idea of what is occurring.

The uploads in this component are hardcoded to do a maximum of 5 files asynchronously, so as not to overload the server when multiple users are uploading.  This is why the time to upload 1500 files would increase linearly compared to 300 files.  You can see in this screenshot from Chrome's Network tab how the start of the upload for the 6th file is delayed until the first 5 have completed.

It is also worth evaluating the performance of uploads against other concurrent web requests on the application, to ensure a user uploading 1500 files does not result in a sort of denial of service attack.

I have not considered this component as an ingestion tool, rather a better user experience for users uploading a few files.  However, that does not mean it could not be a valid scenario.

If you are looking for a change to the component:

Could you confirm if your project requires an ingestion tool that would be required to upload 1500 files at one time in the production application, or whether you are just benchmarking different upload components?

Also, what is your expectation of the performance for the upload, also for the impact on the rest of the application?

I hope this helps clarify things!

Kind regards,

Stuart

Good day Stuart,

Sorry for the late reply. Our project really needs to upload 1500 files at one time and it will be deployed in production.

Our expectation on the performance of the upload would be able to upload 1500 files at less than 30 mins. 

I hope you could help us in our requirement and it will be highly appreciated if this is resolved or would somehow bring us an idea on how to resolve this matter. Thank you very much!!!

Hi Jaybriel,

I could make the maximum parallel uploads configurable instead of hardcoded to 5.  That way you could change it to a value that would suit your scenario.  Would that help?

You may be across this, but I will mention it in case it helps.  A common pattern for handling imports with many uploaded files is to separate the upload and the processing of the file into separate steps.

This way you can upload files and prepare them over a period, then when you are ready, click a button to begin processing them.  The benefits it has are:

* The time required to upload is reduced

* If an upload fails, there is no need to rollback any processing as it has not started yet

* If there is an error during processing, the uploads do not need to be done again

* If the error is related to one of the uploaded files, only that file needs to be reuploaded, not the whole lot.

Additionally, separating the processing of files into batches or if possible, one by one using a process can help protect processing of a particular file from being affected by a failure in another.

Of course, I do not know your specific requirements, so you will have to evaluate these patterns against your specific needs.

I hope this helps!

Kind regards,

Stuart