Hi all,
I want to use OutSystems to process data between two web services. I need to read a large file stream from one service and send it chunk by chunk to another service. My question is: Is it possible to process only a part of the file at once without loading the entire file into memory? Specifically, can I read a chunk of the file, process it, and then send it using a REST API to another service?
I can see the possibility of sending data as a binary parameter, but can I use a stream to read and send it in a loop within the scope of a single request?
Thank you!
Hi Stefan,
Thank you for your answers. I believe the correct answer is that no, there is no native way in OutSystems to handle streams.
Hi Viktar,
in general when working with large files I suggest to use a S3 bucket to store and retrieve it, which gives you byte range support out of the box.
Reading binary data from the database always results in retrieving the file fully into memory. There is no way to read just a range of a binary data from the database.
What you could do (if running O11 on-premise) to store large files in the file system. This component https://www.outsystems.com/forge/component-overview/68/filesystem-o11 allows to create and read binary data files to a filesystem.
You could now write a custom code extension that opens a requested file as a stream from the filesystem, reads only a range of bytes of the stream and returns it to your application.
In your application you could create an exposed REST API GET endpoint that takes the Bytes header with a given range to use with your extension.
Just theoretical.
Thank you for your insights!
My focus is on developing a client web service that consumes a third-party service's API. I don't have the ability to change the APIs themselves, and both the database and filesystem are out of scope for my case. I am reading files from one API into memory, processing and passing them to another API. API's a defined file (binary stream) parameter.
Specifically, I want to know if it's possible to read chunks of a file stream simultaneously and send them to another service without loading the entire file into memory. I’ve searched the documentation but have not found any examples related to handling streams, so I may be missing something.
I appreciate any guidance you can provide!
Viktar
can you give some more details on boths APIs? You get files from a source API. In order to process a file in OS you need the full file. You end up with a processed file in memory that you want to send to a target API. Streaming is often misinterpreted that is why Iam asking for details. For sending a file in multiple chunks you often see a pattern where first a REST call is made to initiate a chunked upload. Then a file is sent in multiple parts and finally a closing REST call is made to conclude the chunked upload.
A binary stream parameter is a single encapuslated sequence of bytes in a request body of a single REST operation. Unlike with WebRTC REST calls are always single request/responses.
Best
Stefan
But in any way, without custom code a chunked upload is not doable :-( At least I cannot think of any way doing that.