Skip to Content (Press Enter)
UserImage.jpg
Rohan J
12Views
2Comments
[CSVUtil] Large Data Set Not able to open the file generated as CSV Binary
Question
Forge
OutSystems 11
Service
csvutil
Service icon
CSVUtil (O11)
Forge asset by Wei Zhu
Application Type
Service

Hi 

Function : ExportRecordList2CSVBinary

RecordList Count around more than 1430951 records (with 40 columns)

The function generate the CSVBinary file but with huge in size around 500MB

I am storing this large file in Network Drive but could not able to open that file. 


While opening the file on Network Drive.

After close of this popup

After convert This sheet shows only 1048576 records only which is max limit for the excel workbook.

Q1: For that much record count it should generate 500Mb size CSV file?

Q2: If Q1 is Yes then How to handle above Scenario. 

 

0
0
16 Oct 2025
Copy Link
2026-03-04 12-08-39
Luiz Henrique Stanqueviski

Vi @Rohan J 

Q1: Can a CSV file with around 1,430,951 rows and 40 columns really be 500 MB in size?

Yes, that is absolutely possible and expected.

The size of a CSV file depends on:

  • Number of cells: 1,430,951 rows × 40 columns = about 57 million cells.
  • Average characters per cell: assuming 8–15 characters on average (including separators, quotes, and line breaks).
    • 57,000,000 × 10 ≈ 570 MB of plain text, which is consistent with the 500 MB you observed.

Additionally:

  • CSV is plain text, meaning no compression.
  • It has no data type optimization (numbers, dates, etc. are stored as text).

Therefore, the file size matches the data volume.

Limitation: Excel only supports 1,048,576 rows

This explains why:

  • The CSV file is correctly generated (1,430,951 rows),
  • But Excel only displays up to 1,048,576 rows, which is the maximum limit for both .xlsx and .csv files opened in Excel.


Q2: How to handle this scenario

Option 1 — Generate an XLSX file with multiple worksheets

  • The .xlsx format supports multiple sheets, each up to 1,048,576 rows.
  • You can therefore:
    • Generate an .xlsx file with two or more sheets (e.g., “Part 1”, “Part 2”, etc.).
    • Split the dataset into chunks of up to 1,048,576 records each.

Advantages:

  • The user can open the file in Excel without issues.
  • The final file is much smaller, since XLSX is ZIP-compressed internally.


Search in the Forge component  of Excel Package (Advanced) or Advanced Excel.

Both support creating multiple worksheets dynamically.

Option 2 — Generate multiple CSV files (simpler, no Excel)

Split the export into multiple CSVs (e.g., 1 million rows each):


  • Data_1.csv (rows 1–1,000,000)
  • Data_2.csv (rows 1,000,001–1,430,951)

This reduces file size and improves manageability.

Another possible approach is to use tools such as DBeaver or similar database viewers to open the CSV file as a database table. This allows you to query, filter, and analyze the data directly without relying on Excel, which is ideal for handling very large datasets efficiently.

This answer was partially generated by ChatGPT and refined by me through analysis and evaluation of the AI’s response.

I hope help you!

1
0
17 Oct 2025
Copy Link
2025-12-29 06-17-15
Deepak Raj M

Hi @Rohan J In download widget .xlsx replace .csv , if it is fine large data means you can use integration studio and write code in .net and convet it to binary and u can download large data, if more detail required contact me

0
1
17 Oct 2025
Copy Link
Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.