1.0.1 Update: denote in red

Notes:

If you encounter a 'Server Request Timeout' exception (especially when uploading files larger than 200MB), consider increasing the 'Server Request Timeout' property of the 'MultipartUpload_CW/Client Actions/MultipartUpload/CompleteMultipartUpload/CompleteMultipartUpload' Action node.

Dependencies

• OutSystems UI (built-in)
• Binary Concat
• Amazon Simple Storage Service (S3)

Architecture

MultipartUpload_CW (Core Widget) → MultipartUpload_API (Exposed API) → MultipartUpload_CS (Core Service) → MultipartUpload_AmazonS3_ext (Extension)

Detailed Functionalities

1. MultipartUpload_CW: Exposes core functionalities of the component.

• Block
  • MultipartUpload: Upload widget enables uploading multiple large files to OutSystems' database at once.
  • MultipartUpload_CSV: Upload widget specialized for uploading large CSV files efficiently by splitting them into 10MB chunks (ending at line boundaries), appending headers to each chunk, and uploading these as individual parts.
    • Ideal for data import scenarios, allowing batch processing of CSV data for reliability and performance.
  • MultipartUpload_S3: Same functionality as MultipartUpload block but upload files to AWS S3.
    • Supports adding prefixes to S3 objects.
  • MultipartUpload_S3_CSV: Same functionality as MultipartUpload_CSV block but upload files to AWS S3.
    • Supports adding prefixes to S3 objects.
• Client Actions
  • FileDownload: Download a file locally using REST API instead of OutSystems' Download Action given the File UID. Improved upload speed with progress tracking by the browser.
• Server Actions
  • FileDelete: Delete a file from the database given the File UID.
  • FileGet: Get the file information & binary from the database given the File UID.

2. MultipartUpload_API: REST API for file uploading & downloading.

• Integrations/REST/v1: Expose methods for single & multi-part file uploading, enables file download through REST API.
  • Custom authentication: Checks if the File UID exists in the request URL and if that File UID exists in the File table.
• Service Actions
  • GetPreSignedUrl_Download
  • GetPreSignedUrl_Multiple
  • GetPreSignedUrl_Single

3. MultipartUpload_CS: Database tables and Server Actions to support file uploading, downloading, and concatenating binary chunks.

• Database tables
  • File: File information (name, size, content-type).
  • FileBinary: File actual binary after concatenating chunks/parts.
  • FilePart: File's binary chunks/parts.
• Server Actions
  • AbortMultipartUpload
  • CompleteMultipartUpload
  • CreateMultipartUpload
  • New actions in S3 folder to support multi-part upload with AWS S3
• Processes
  • DeleteFileParts: Invoke after concatenating file in CompleteMultipartUpload action.

4. MultipartUpload_AmazonS3_ext: Cloned from 'Amazon Simple Storage Service (S3)' Forge asset to add custom methods

• Server Actions
  • S3_GetPreSignedUrl_MultipartUpload
    • Enhanced to accept two additional input parameters: UploadId and PartNumber.
    • Enables direct multipart uploading to AWS S3 using Presigned URLs, allowing clients to upload parts directly to S3 without routing data through OutSystems servers.
    • This approach avoids the extra memory consumption associated with the standard Object_UploadPart action from AmazonS3_IS, which loads file parts into OutSystems server RAM before forwarding to S3.

AWS S3 Setup

1. In AWS Console:

• Create a S3 bucket.
• In the 'Permissions' tab, add 'Cross-origin resource sharing (CORS)' settings:

[

    {

        "AllowedHeaders": [

            "*"

        ],

        "AllowedMethods": [

            "HEAD",

            "GET",

            "PUT",

            "POST",

            "DELETE"

        ],

        "AllowedOrigins": [

            "*"

        ],

        "ExposeHeaders": [

            "ETag"

        ]

    }

]

• Create an IAM user with permission to access the S3 bucket.
  • Remember to save the IAM user credentials.

2. In OutSystems Service Center:

• Go to module 'MultipartUpload_CS'.
• In the 'Site Properties' tab, provide these info:
  • Mandatory
    • AccessKeyId: IAM user credential.
    • BucketName: Name of the bucket you created in the previous step.
    • Region: For example, 'APNortheast1' or 'USEast1'.
    • SecretAccessKey: IAM user credential.
  • Optional
    • PresignedUrlExpiresIn: Time in seconds that the Presigned Urls are going to expire, defaults to 1800 seconds or 30 minutes.

Dependencies

• OutSystems UI (built-in)
• Binary Concat
• Amazon Simple Storage Service (S3)

Architecture

MultipartUpload_CW (Core Widget) → MultipartUpload_API (Exposed API) → MultipartUpload_CS (Core Service) → MultipartUpload_AmazonS3_ext (Extension)

Detailed Functionalities

1. MultipartUpload_CW: Exposes core functionalities of the component.

• Block
  • MultipartUpload: Upload widget enables uploading multiple large files to OutSystems' database at once.
  • MultipartUpload_CSV: Upload widget specialized for uploading large CSV files efficiently by splitting them into 10MB chunks (ending at line boundaries), appending headers to each chunk, and uploading these as individual parts.
    • Ideal for data import scenarios, allowing batch processing of CSV data for reliability and performance.
  • MultipartUpload_S3: Same functionality as MultipartUpload block but upload files to AWS S3.
    • Supports adding prefixes to S3 objects.
  • MultipartUpload_S3_CSV: Same functionality as MultipartUpload_CSV block but upload files to AWS S3.
    • Supports adding prefixes to S3 objects.
• Client Actions
  • FileDownload: Download a file locally using REST API instead of OutSystems' Download Action given the File UID. Improved upload speed with progress tracking by the browser.
• Server Actions
  • FileDelete: Delete a file from the database given the File UID.
  • FileGet: Get the file information & binary from the database given the File UID.

2. MultipartUpload_API: REST API for file uploading & downloading.

• Integrations/REST/v1: Expose methods for single & multi-part file uploading, enables file download through REST API.
• Service Actions
  • GetPreSignedUrl_Download
  • GetPreSignedUrl_Multiple
  • GetPreSignedUrl_Single

3. MultipartUpload_CS: Database tables and Server Actions to support file uploading, downloading, and concatenating binary chunks.

• Database tables
  • File: File information (name, size, content-type).
  • FileBinary: File actual binary after concatenating chunks/parts.
  • FilePart: File's binary chunks/parts.
• Server Actions
  • AbortMultipartUpload
  • CompleteMultipartUpload
  • CreateMultipartUpload
  • New actions in S3 folder to support multi-part upload with AWS S3
• Processes
  • DeleteFileParts: Invoke after concatenating file in CompleteMultipartUpload action.

4. MultipartUpload_AmazonS3_ext: Cloned from 'Amazon Simple Storage Service (S3)' Forge asset to add custom methods

• Server Actions
  • S3_GetPreSignedUrl_MultipartUpload
    • Enhanced to accept two additional input parameters: UploadId and PartNumber.
    • Enables direct multipart uploading to AWS S3 using Presigned URLs, allowing clients to upload parts directly to S3 without routing data through OutSystems servers.
    • This approach avoids the extra memory consumption associated with the standard Object_UploadPart action from AmazonS3_IS, which loads file parts into OutSystems server RAM before forwarding to S3.

AWS S3 Setup

1. In AWS Console:

• Create a S3 bucket.
• In the 'Permissions' tab, add 'Cross-origin resource sharing (CORS)' settings:

[

    {

        "AllowedHeaders": [

            "*"

        ],

        "AllowedMethods": [

            "HEAD",

            "GET",

            "PUT",

            "POST",

            "DELETE"

        ],

        "AllowedOrigins": [

            "*"

        ],

        "ExposeHeaders": [

            "ETag"

        ]

    }

]

• Create an IAM user with permission to access the S3 bucket.
  • Remember to save the IAM user credentials.

2. In OutSystems Service Center:

• Go to module 'MultipartUpload_CS'.
• In the 'Site Properties' tab, provide these info:
  • Mandatory
    • AccessKeyId: IAM user credential.
    • BucketName: Name of the bucket you created in the previous step.
    • Region: For example, 'APNortheast1' or 'USEast1'.
    • SecretAccessKey: IAM user credential.
  • Optional
    • PresignedUrlExpiresIn: Time in seconds that the Presigned Urls are going to expire, defaults to 1800 seconds or 30 minutes.

0.1.0 Update: denote in red

Dependencies

• OutSystems UI (built-in)
• Binary Concat

Architecture

MultipartUpload_CW (Core Widget) → MultipartUpload_API (Exposed API) → MultipartUpload_CS (Core Service)

Detailed Functionalities

1. MultipartUpload_CW: Exposes core functionalities of the component.

• Block
  • MultipartUpload: Upload widget enables uploading multiple large files at once.
  • MultipartUpload_CSV: New block specialized for uploading large CSV files efficiently by splitting them into 10MB chunks (ending at line boundaries), appending headers to each chunk, and uploading these as individual parts.
    • Ideal for data import scenarios, allowing batch processing of CSV data for reliability and performance.
• Client Actions
  • FileDownload: Download a file locally using REST API instead of OutSystems' Download Action given the File UID. Improved upload speed with progress tracking by the browser.
• Server Actions
  • FileDelete: Delete a file from the database given the File UID.
  • FileGet: Get the file information & binary from the database given the File UID.

2. MultipartUpload_API: REST API for file uploading & downloading.

• Integrations/REST/v1: Expose methods for single & multi-part file uploading, enables file download through REST API.
• Service Actions
  • GetPreSignedUrl_Download
  • GetPreSignedUrl_Multiple
  • GetPreSignedUrl_Single

3. MultipartUpload_CS: Database tables and Server Actions to support file uploading, downloading, and concatenating binary chunks.

• Database tables
  • File: File information (name, size, content-type).
  • FileBinary: File actual binary after concatenating chunks/parts.
  • FilePart: File's binary chunks/parts.
• Server Actions
  • AbortMultipartUpload
  • CompleteMultipartUpload
  • CreateMultipartUpload
• Processes
  • DeleteFileParts: Invoke after concatenating file in CompleteMultipartUpload action.

Dependencies

• OutSystems UI (built-in)
• Binary Concat

Architecture

MultipartUpload_CW (Core Widget) → MultipartUpload_API (Exposed API) → MultipartUpload_CS (Core Service)

Detailed Functionalities

1. MultipartUpload_CW: Exposes core functionalities of the component.

• Block
  • MultipartUpload: Upload widget enables uploading multiple large files at once.
• Client Actions
  • FileDownload: Download a file locally using REST API instead of OutSystems' Download Action given the File UID. Improved upload speed with progress tracking by the browser.
• Server Actions
  • FileDelete: Delete a file from the database given the File UID.
  • FileGet: Get the file information & binary from the database given the File UID.

2. MultipartUpload_API: REST API for file uploading & downloading.

• Integrations/REST/v1: Expose methods for single & multi-part file uploading, enables file download through REST API.
• Service Actions
  • GetPreSignedUrl_Download
  • GetPreSignedUrl_Multiple
  • GetPreSignedUrl_Single

3. MultipartUpload_CS: Database tables and Server Actions to support file uploading, downloading, and concatenating binary chunks.

• Database tables
  • File: File information (name, size, content-type).
  • FileBinary: File actual binary after concatenating chunks/parts.
  • FilePart: File's binary chunks/parts.
• Server Actions
  • AbortMultipartUpload
  • CompleteMultipartUpload
  • CreateMultipartUpload
• Processes
  • DeleteFileParts: Invoke after concatenating file in CompleteMultipartUpload action.