[ReactFilePondUpload] Where is the data stored?

Forge Component
(5)
Published on 27 May by Stuart Harris
5 votes
Published on 27 May by Stuart Harris

Hi,

I'm want to use this component but I have a hard time figuring out where the files are between upload and download (with token). The only reference I have found is filepond-root and I think this is a folder on the servers Temp storage but I can't verify that due to being on a hosted solution.

Can you help me with this? And if it is a local folder, how does this work in a multi-server environment?

Hi Vincent,

Thank you for asking the question.  The files are actually stored in the database, primarily to avoid multi server/load balancing problems.  There are server actions to retrieve the files which abstract the location of storage.

Once you have retrieved the tokens with FilePondUpload_GetTokens() pass them to the server and call server action FilePondUpload_GetUploadedFiles(UploadTokens).

Also make sure you clear the tokens on the client side with FilePondUpload_Reset().

There are some instructions in the documentation on the forge component.  Also the sample gives a good example of how to use it.  So if you have not seen these yet, I encourage you to have a look.  They should help out.

https://www.outsystems.com/forge/Component_Documentation.aspx?ProjectId=7405&ProjectName=reactfilepondupload

I hope this answers your question.

Kind regards,

Stuart

Hi Stuart,

My usecase is uploading very large files (between 250MB and 500GB (yes, giga)). Storing these in the database first is not really an option since we don't have the database space available to recieve multiple files at once and we want to store the files somewhere else entirely.

I looked into FilePond and noticed that you need to connect to a server that can recieve these files (did that before creating this thread). I didn't find a rest endpoint in your code but since I missed the database part this could also be an oversight. But is it possible to utilize this server configuration to send data to my own rest point that can then handle the data by storing the files at the correct location (azure storage accounts of SharePoint)? If it's possible, how can I do that?

Kind regards,

Vincent

Hi Vincent,

Thank you for explaining your use case, that really helps.

For the size of files you are dealing with, a different scenario is certainly required.  I suspect the upload will timeout before the files are fully uploaded.  Your scenario requires a different forge component specifically designed for large file upload. As you point out, a different storage mechanism is appropriate, such as Azure Blob storage, as well as ensure the timeout is large enough to support the time required to upload; either that or chunk the uploads.  But then you need a storage mechanism that can support partial uploads.

I know this is not the answer you are looking for, but I hope this gives you some further direction.

Kind regards,

Stuart


Hi again Vincent,

I also wanted to add another potential solution.  This would only work in the case that trusted users are performing the uploading.

For files this size, if the scenario works for you, it would be beneficial to avoid using the web application as a middleman during the upload.  This has the advantage that you do not have to expose an upload endpoint with a long timeout and a large maximum content size that could be used as a denial of service attack.

The trusted users could upload the file direct to storage such as Azure Blob or AWS S3 separately from the web application, then providing the web application a url or identifier for the chosen storage system via a REST API call.

Potentially a powershell script or a .NET console application would work.  In the case of using powershell and Azure Blob storage you could use AzCopy or to AWS S3 AWS Tools (also here).

Also, here is a link for powershell SharePoint upload

Then call the REST API you have exposed in your web application to post the location of the file.  Both the REST API and the storage mechanism will require authentication, which is why the solution would be limited to trusted users, as well as having to install something on their machine.

Just an alternative idea of how you could solve the problem.  However, it does make alot of assumptions about the trusted nature of your users doing the uploaded.

I hope this is helpful.

Kind regards,

Stuart

Hi Stuart,

Thank you for your answers. I'll continue my search for something suitable (either inside or outside the Forge :)). 

edit: To answer your suggestion about uploading directly into the storage account. This is a possibility that we looked into but the range of people using our application and their IT knowledge is so diverse that we don't want to go into this path unless absolutely necessary.