AsyncFileProcessing

Stable Version 1.0.1 (OutSystems 9.1)
Published on 25 July 2016 by 
Created on 04 July 2016
Details
Asynchronous large file processing patterns
Read More
There are several ways to address the execution of batch jobs with OutSystems Platform, either by using the built-in Timers functionality or by extending the Platform with BPT (Business Processing Technology).
 
They both support asynchronous logic execution, but while Timers are a single thread execution usually for time bound operations — within the minutes time range —, the BPT Processes allow parallel execution, with multiple atomic activities, where each process can take hours or even days, out of the box.
 
You can use standalone Timer or BPT processes, or you can use them in conjunction.
 
When used in conjuction, you define one or the other as the orchestrator. Each of these solutions may be more appropriate for a given scenario, so you have to ponder their pros and cons and make your choice accordingly.
 
1. Timer: reads a certain amount of lines and processes chunks at each run.
Pros
  • supports long runs to process files
  • pooling mechanism to detect new files
Cons
  • only handles one file at a time and doesn't support parallelism
2. BPT: triggers a single process that handles multiple chunks during each run
Pros
  • handles multiple files at a time
  • supports parallelism to process chunks of files
Cons
  • doesn't support pooling mechanisms to detect new files (dependent on an external service triggering the BPT process)
  • no long runs
  • adds a certain degree of load on the database, due to the BPT engine that needs to be purged
3. Timer with BPT: triggers several parallel processes for each chunk, which is appropriate when you don't have to keep a sequence between lines.
Pros
  • supports pooling mechanisms to detect new files
  • parallelism to process chunks of files
Cons
  • cannot handle multiple files at a time
  • cannot handle long runs.
4. BPT with Timer: BPT orchestrates the asynchronous process execution, by using Timers.
Pros
  • handles multiple files at a time
  • supports long runs
Cons
  • no parallelism to process chunks of files
  • no pooling mechanism to detect new files (an external service is needed to trigger the BPT process)
  • additional load on the database
What’s new (1.0.1)

Fixed issue on Batch_BPT and Batch_TimerWithBPT examples, when changing file chunk status (FileChunk_SetStatus action)

Reviews (0)
Category
Samples & How-tos
Support Options
This component is not supported by OutSystems. You may use the discussion forums to leave suggestions or obtain best-effort support from the community, including from Ricardo Gonzaga who created this component.
Dependencies
See all 1 dependencies
Requirements
Platform:
9.1.0.6
Database:
All
Stack:
All
Component Consumers
No consumers yet.
Weekly Downloads