92
Views
3
Comments
ODC: Merging Chunks of Large Binary Files Exceeding Input Payload Limit
Question
Application Type
Reactive

I am working on an application where I need to handle large Excel files (up to 40 MB). Here’s how I’m processing the files:

  1. Upload and Chunking:

    • The file is uploaded via the browser.
    • It is split into 5 MB chunks and saved into the database.
  2. Processing with Timer:

    • A Timer fetches these chunks from the database and processes them asynchronously.
    • The Timer calls a .NET extension to merge the chunks back into a single binary file using a method similar to the BinaryConcat function in the Binary Concat Forge component.
  3. Issue:

    • When I pass all the chunks to the .NET extension method, I receive the following error:

      name: 'OutSystems.Application.ErrorHandling.ExtensionException', message: 'Input payload is too large (33.33MB), maximum allowed is 5.5MB.', stack: 'OutSystems.Application.ErrorHandling.ExtensionException'

    • This indicates that the input payload exceeds the OutSystems limit of 5.5 MB.

Questions:

  1. Is there any way to increase the maximum allowed payload size in OutSystems ODC for server-side processing?
  2. If not, what is the best approach to handle this requirement? My main goal is to reassemble the file efficiently to process it asynchronously.
  3. Is there another recommended method in OutSystems to handle such large files?

Additional Details:

  • The chunks are stored in the database as binary data.
  • The final merged binary file needs to be processed further (parsing the Excel data).
  • This process is triggered and executed entirely on the server-side (via a Timer).

Looking forward to suggestions and best practices for managing this scenario. Thank you!

2024-07-31 11-32-34
ndy

Dear @Mayank Dharmpurikar .

Implement an external File Processing microservice to handle file merging and processing.

1. OutSystems uploads the chunks to a cloud storage service (e.g., AWS S3, Azure Blob Storage).

2. Metadata for chunks (e.g., file URLs) is sent to a microservice.

3. The microservice merges and processes the file, returning the result to OutSystems.

2019-01-07 16-04-16
Siya
 
MVP

Please have a look at the "Use with large binary files" of the documentation at https://success.outsystems.com/documentation/outsystems_developer_cloud/building_apps/extend_your_apps_with_external_logic_using_custom_code/external_libraries_sdk_readme/ . This talks about the issue you are facing and some work arounds.

2025-06-28 01-52-12
Marco Mateo

Hi,

I am also experiencing the same issue using AWS SDK for multi part upload in S3 bucket.

We are already slicing the binary into 5MB (5242880) but when we pass it to server action (external logic), it throws an error Input payload is too large (6.67MB), maximum allowed is 5.50MB

Any idea for workaround?

At least any document or info on how ODC compute for the Input Payload?

Thank you!

FYI. The minimum file that AWS accepts is 5MB. Hence the reason why I am chunking it to 5MB. Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.