With version 10.0.800, OutSystems introduced Light BPT as an addition to the traditional timers and ‘standard’ BPT. Light BPT allows for fast processing of large amounts of transactions thanks to its Fire & Forget methodology. This feature will help many organizations move their data migration projects into the fast lane.

Light BPT can make the Cool Data Mover – a tool for automated data migration in OutSystems - super effective cutting the processing time by a factor of 20. How can you make optimal use of this supercharger and bring your data migration game to the next level?

What Is Light BPT?

Before the introduction of Light BPT there were only timers and ‘standard’ BPT (Business Process Technology) available for asynchronous processes in OutSystems. Both are excellent tools but not intended for fast processing of large amounts of transactions.

Light BPT is specifically designed to handle large volumes of transactions. Because the platform records no process history, there is virtually no overhead involved, which is a great benefit over standard BPT.

Thanks to Light BPT's Fire & Forget methodology, we now have an engine strong enough to substantially increase the processing speed of transactions.

Some of the characteristics of Light BPT:

• No activity logging overhead;
• Scheduler service;
• Batch scheduling;
• Schedules ~4-10 threads per tick (100 nanoseconds);
• Default max 20 threads per front-end;
• Factory configuration can be used to increase threads;
• Not always an even load distribution;
• Only 1 automatic activity.

How Light BPT Enhances the Cool Data Mover

The main task of the Cool Data Mover is to export and import large quantities of records from and to OutSystems databases. This is doable with standard BPT as long as the amount of data is limited.

For larger quantities of data, however, standard BPT is inadequate. It’s simply not fast enough. The processing time for data exports and imports grows quickly as the amount of data increases. Even after many optimizations, the Cool Data Mover could not get past processing 300 records per second with standard BPT.

Light BPT makes it possible to start multiple processes simultaneously and to process them quickly. This enabled to increase the performance of the Cool Data Mover by a factor of 20 and reduce the processing time of large amounts of data from several hours to mere minutes.

An Engine Management System for Light BPT

Understandably, simply replacing the engine of your application with Light BPT is not enough. To make optimal use of Light BPT, applications need to be adjusted. It is essential to split the total workload into multiple small tasks. Small tasks have the advantage of being manageable, they require few resources, only use the database for a short period of time, and can be implemented in a Light BPT process.

The downside is that a mechanism is needed to coordinate all these small tasks. Compare it to a car with Light BPT being a powerful engine: you need an engine management system to control fuel intake, and excellent brakes to keep the power source in check. We've implemented a standard BPT process as the engine management system. This process manages the flow of complex data migrations. It doesn't have to be super fast, as long as it generates enough work to keep the Light BPT engine running continuously.

Of course, tasks should not become too small. That would make the overhead too big. Just like a car, it needs a bit of fine-tuning here and there to make the whole process function optimally and smoothly. That is, look for the optimal size of tasks.

How to Get More Out Of Light BPT

Light BPT can also be tuned to deliver more horsepower. We can do that by making more servers responsible for the processing. By default, a single server will handle a maximum of 20 processes simultaneously. Increasing this number is one option, but it is much more efficient to scale servers to make the engine run faster.

Scaling the database by adding servers is often impossible. You can only scale to a certain extent by adding more resources to the database server because normally, the database would become the bottleneck.

Therefore, good brakes are needed to slow down in time before the database collapses under the pressure of all the transactions running simultaneously. Before the use of Light BPT the database ‘only’ had to process hundreds of transactions per second, with Light BPT it could be thousands of transactions.

How Does It Work?

Our BPT engine management system - called the dispatcher - contains the logic to decide which tasks can be done next. It dispatches all tasks that are required without worrying about when they will be executed.

To avoid overloading the environment, the tasks are not dispatched directly as Light BPT processes but as a second BPT process, the mediator. The mediator’s main function is to regulate the workload that is being transferred as Light BPT.

It measures the amount of work that is being executed, determines whether there is spare capacity, and submits new tasks when possible. This means being in control of how much work will be processed simultaneously and slowing down (braking) when necessary. This mechanism makes it possible to process large amounts of data during production hours, although it might progress a bit slower.

We need the right information at the right time to be able to brake effectively. This requires monitoring to allow automated and manual decisions throughout the entire process.

A Crash! Now What?

Not everything will always run smoothly. Many factors can affect the process. For the Cool Data Mover it could be data inconsistencies, but there are also external influences, such as network interruptions, database issues, etc.

Because Light BPT uses a Fire & Forget method, there is no standard insight into the results of the processing. The platform also does not register any information from Light BPT processes.

It is therefore important to set up a Monitoring and Logging mechanism so that, in the event of a breakdown, you have the correct information for a thorough analysis. CoolProfs can help set this up so your data migration activities will always run smoothly.

To learn more about, visit Migration as a Service page in the OutSystems Forge.

In collaboration with: