42
Views
4
Comments
Download/Export large records CSV file
Question
Application Type
Reactive

HI Community,

There is the use case scenario where I wanted to download/extract a large amount of records (like 15Lakhs or 25Lakhs) in to only one CSV file. Since excel file has limitation to numbers of rows but that is not the case with csv file. 

  • Don't want to store that file into network drive.
  • Don't want to export in multiple csv's file.
  • Don't want to increased the server time out drastically.  

Let me know any pointers what should be the best approach so that there should not be connection time out issues.

2024-12-18 12-00-57
Gustavo Gonçalves

Hey! I’ve faced a similar situation before when exporting large datasets, and the best balance I found was to handle the generation asynchronously in the background.

Instead of building the entire CSV during the user’s request (which will almost always hit a timeout), you can:

  1. Trigger a background job (using a Timer or a BPT Process) when the user clicks “Export CSV”.

  2. In that process, fetch data in batches (something like 100k rows per iteration) and build the CSV progressively in memory.

    • You can append each batch to a BinaryData variable or use a TextBuilder to handle the text efficiently.

  3. Once all batches are processed, store the CSV temporarily (for example, in the app’s temporary storage or as a transient binary in the database — not in a network drive).

  4. When it’s done, just expose a download link or send a notification to the user saying the export is ready.


    Hope this helps! 
    GG.

UserImage.jpg
Rohan J

Hi @Gustavo Gonçalves 

I did the same as you mentioned above.

Generated binary file by csv util component and store that binary data file in a Outsystems table.

Binary data file size is almost 700MB (which is vary large in size)

I gave the download link to download this 700 MB file data which is ultimately gives time out error in the UI.

Definitely, We need another approach for this use case. 

2019-04-13 08-16-46
Pawan Kumar Sharma

Hi @Rohan J ,

Trying to serve a 700MB binary file directly through the OutSystems UI via a download link is likely to cause timeouts, memory issues, or performance degradation, especially in browser-based environments.

  • Best suggestion that I can think of is to Store the File in External Storage (e.g., Azure Blob, AWS S3, SharePoint) instead of storing the binary in an OutSystems entity:
  • Upload the file to a cloud storage service.
  • Store only the URL or metadata in OutSystems.
  • Use pre-signed URLs for secure, time-limited access.

Benefits:

  • Offloads storage and bandwidth from OutSystems.
  • Supports large file downloads efficiently.
  • Avoids UI timeouts.

If you must keep the file in OutSystems (not recommended for 700MB):

  • Break the file into smaller chunks.
  • Store each chunk in a separate record.
  • Reconstruct the file on the client or server side.

But this is complex and not ideal for UI download scenarios.

2025-07-22 10-30-27
Mandar Deshpande

Hi @Rohan J 

The most common enterprise solution in this case could be to compress the file. 

CSV size is driven by row width, not row count alone. Compression ratio for CSV usually yields 60–85% reduction. So, around 1.2 GB CSV → 300–400 MB ZIP. So, the steps would be: 

  • Generate CSV
  • ZIP it (server-side)
  • Store ZIP binary
  • User downloads .zip

So, there would be still one file, faster download and less DB stress.

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.