AsyncFileProcessing

Stable Version 1.0.1 (OutSystems 9.1)
Published on 25 July 2016 by 
Created on 04 July 2016
Details
Asynchronous large file processing patterns
Read More
There are several ways to address the execution of batch jobs with OutSystems Platform, either by using the built-in Timers functionality or by extending the Platform with BPT (Business Processing Technology).
 
They both support asynchronous logic execution, but while Timers are a single thread execution usually for time bound operations — within the minutes time range —, the BPT Processes allow parallel execution, with multiple atomic activities, where each process can take hours or even days, out of the box.
 
You can use standalone Timer or BPT processes, or you can use them in conjunction.
 
When used in conjuction, you define one or the other as the orchestrator. Each of these solutions may be more appropriate for a given scenario, so you have to ponder their pros and cons and make your choice accordingly.
 
1. Timer: reads a certain amount of lines and processes chunks at each run.
Pros
  • supports long runs to process files
  • pooling mechanism to detect new files
Cons
  • only handles one file at a time and doesn't support parallelism
2. BPT: triggers a single process that handles multiple chunks during each run
Pros
  • handles multiple files at a time
  • supports parallelism to process chunks of files
Cons
  • doesn't support pooling mechanisms to detect new files (dependent on an external service triggering the BPT process)
  • no long runs
  • adds a certain degree of load on the database, due to the BPT engine that needs to be purged
3. Timer with BPT: triggers several parallel processes for each chunk, which is appropriate when you don't have to keep a sequence between lines.
Pros
  • supports pooling mechanisms to detect new files
  • parallelism to process chunks of files
Cons
  • cannot handle multiple files at a time
  • cannot handle long runs.
4. BPT with Timer: BPT orchestrates the asynchronous process execution, by using Timers.
Pros
  • handles multiple files at a time
  • supports long runs
Cons
  • no parallelism to process chunks of files
  • no pooling mechanism to detect new files (an external service is needed to trigger the BPT process)
  • additional load on the database
What’s new (1.0.1)

Fixed issue on Batch_BPT and Batch_TimerWithBPT examples, when changing file chunk status (FileChunk_SetStatus action)

Reviews (0)
Category
Samples & How-tos
Support Options
This component is not supported by OutSystems. You may use the discussion forums to leave suggestions or obtain best-effort support from the community, including from Ricardo Gonzaga who created this component.
Dependencies
See all 1 dependencies
Requirements
Platform
9.1.0.6
Database
All
Stack
All
Component Consumers
No consumers yet.
Weekly Downloads 
Related Components
Dynamic Forms
Huarlem Lima
Component to create dynamic forms with validation, tips and dependent fields. Very useful when the user needs to create his form, create polls, surveys, quiz and checklists. You can sort the fields by dragging and dropping.
724
How To - Lazy load
Francisco Menezes
Learn how to handle heavy computation screens by quickly rendering the main content structure, while sub-content is progressively loaded 
948
Business Process Lesson Resources
Scott Karabin
The Business Process Lesson Resources are typical Outsystem's elements that have been pre-built for the Modeling Business Process lessons. Loading these pre-built elements will allow the focus of the lessons to be on the example business process.
3246
More from Ricardo Gonzaga
Chat
Hélio Dolores
Mobile and Inline Chat for OutSystems' Applications (uses node.js)
845
IdP
Rui Barbosa
IdP Connector is a generic federated identity provider (IdP) connector, i.e., the IdP Connector its an IdP Service Provider. It allows your OutSystems applications to integrate with single sign-on (SSO) provided by most of the commercial IdP companies. With this integration when the users access an OutSystems application, they are redirected to a web page (known as the enterprise's login manager) where they are prompted to enter their enterprise user name and password. Upon verification of the user’s login, the enterprise identity provider informs OutSystems application of the verified identity for the user who is logging in, and the user is redirected back to the portal website. Also supports SSO from OutSystems mobile apps, alongside with IdP Mobile connector. Tested with Okta, Azure, ADFS, PingOne and OneLogin
1205
OnTrack ECT Connector
Ricardo Gonzaga
Integrate user feedback, submitted via App Feedback (formerly know as ECT), directly into OnTrack projects.
126