Light Processes Under The Hood
Discussion

Hello developers!

 

While messing around with Light Processes, I've realized that there is a lack of information on how it works, so I've decided to give some tips.


You can already find some information on how to activate and on which conditions a BPT Process will be triggered as a Light Process:

But there is more relevant information that in my opinion we all should know.


Useful information / tips:

  1. Light Processes will skip the execution of callback actions (like "On Process Start")

  2. Light Processes won't allow you to add an Output parameter to the Process (if you do, you'll have an internal error on the compiling step while deploying)

  3. By default, Light Processes have 20 threads per front-end, this threads are separate from BPT threads (normal BPT have 10 threads by default).

  4. If an unhandled exception occurs during the execution of a Light Process, the Scheduler will pick that Light Process again moments later (the number of times it failed to execute will increase how much in the future that process will be picked again, until it reaches a point that it will be picked once a day)

  5. Light Processes are executed on its own transaction (session variables and GetUser() will be empty unless you define them during its execution)

  6. Light Processes execution is asynchronous (the transaction responsible for triggering the Light Process will not be waiting for the execution of the process, the process will run on the background)


When to use?

  • Light processes are intended to be used for background processing

  • Replace timers that don’t require scheduling by Light Processes

  • Replace normal BPT Processes only used for background processing containing only 1 automatic activity

    • Even if the normal BPT Process has more than 1 automatic activity, it might be the case that the main workflow (for the 80% of the cases) could be made with a Light Process and only on the exception scenarios

    • For the exception scenarios a normal BPT would be launched via “LaunchProcess” action

    • This could alleviate the server load on the normal BPT Process threads and use the more performant 20 threads from the Light BPT

    • Light Processes don’t create entries on BPT tables, which means less entries on tables Activity, Activity_Output, Process, Process_Input and Process_Output


How does Light Process actually works under the hood?

  1. When defining an Entity “Expose Process Events = True”, the deployment will create an SQL event table containing information relevant for the following triggers

    • The trigger (mentioned further) will look at this table to know which ProcessDefinition will be used and if is Light Process or normal BPT process


  2. After publish, the deployment step will create an SQL table to handle the events over the “File_Event” user table


  3. Create a BPT Process and define the “Launch On” to “Create <your entity>”

    • When you do this, during the deployment, a new record will be added to the event table OSEVT_xcd_File_Event mentioned under the point 2

      1. _PROCESS_DEF_ID = 52 corresponds to the process Background_ProcessFile

      2. _IS_LIGHT is set to 1 because this BPT Process matches all the conditions to be a Light Process

  4. A Trigger will also be created for the normal user table (in this case for the File_Event)

  5. When a record is inserted in the user table, the trigger will:

    • Query the event table OSEVT_xcd_File_Event to fetch the _PROCESS_DEF_ID

    • Insert on the system SQL table ossys_BPM_Event (next point will explain this table) the fetched information together with the Entity_Record_Id which identify the Primary Key from the user table File_Event



  6. Scheduler is constantly pinging the system table ossys_BPM_Event for a record with Next_Run <= Current Date, once found, it allocates a thread to handle that event

    • The running thread mentions the Event id 10 which corresponds to the record being picked from the ossys_BPM_Event

  7. In case an Unhandled Exception occurs during the execution, the table column ossys_BPM_Event.Error_Count will increase by 1 and the ossys_BPM_Event.Next_Run will be set to moments later

    • The number of times it failed to execute will increase how much in the future the Next_Run will be set, until it reaches a point that it will be set to be picked once a day

To add to that, I think the normal BPT has a timeout of 5 minutes, where for light BPT it is 3 minutes.

Nice article Cláudio

Champion

Question on this,

What happens to the record in ossys_BPM_Event if a process has finished correctly? (eg actually reaching an end node).

Currently running in a problem where my BPT gets an exception (timeout or unable to fetch file from server) and performs a rollback. However earlier in this BPT I perform a call to an external system and keep track of this in a seperate Entity in Outsystems, this is then again reverted by the rollback...

Currently I'm solving this by using the CommitTransaction server action after sending the message  to the external system and logging that fact.

I was just thinking that maybe it would be possible to restart the BPT instead of going to an end node, or perhaps starting it with some code by doing stuff to the ossys_BPM_Event table (thought I can imagine this is not recommended), else I would have to create a new process for the exact same record.

Any idea's? Or is my solution to use CommitTransaction the best I can make of this?

Hi Joey,


If the transaction is finished successfully, the record on the "ossys_BPM_Event" table will be removed.

Only if the transaction finishes with an unhandled exception, the record will be kept in the "ossys_BPM_Event" with the "Error_Count" + 1 and "Next_Run" postponed.


If your record is being removed, that's probably because in your code you are catching the exception, and in that case you are handling the exception within your code and the transaction is considered to be executed successful.


If you want it to automatically retry when an exception occurs, then you should avoid Exception Handlers catching "All Exceptions" or any other than "User Exceptions", that way, once an exception is raised, the transaction will immediately stop and the "ossys_BPM_Event.Next_Run" will be postponed to run again.


In your specific case, if I understood correctly, in the same transaction, you are making 2 external calls, since they are external, I assume each one is running on it's own transaction, so if the 1st external call is successful and the 2nd external call raise an exception, your BPT transaction cannot rollback the 1st call because it was executed on a separate transaction.


Hope this information can help.

Champion

Hey Cláudio,

The external calls are made to a system not Outsystems.
I'm logging in Outsystems wether I already made a call to that system for this record (sending duplicates to this system will cause problems). 

Currently, after this logging, I use the CommitTransaction action, so even if the process gets an exception, on the next retry I do not resend data to the external system.

I was looking for a way where I do not have to use CommitTransaction....
Only thing I can think of is catching the exception (without aborting the transaction), perform CommitTransaction and throw another exception to let the BPT restart itself.

Hi Joey,


In your case is totally fine to use the CommitTransaction to flag something which should not be sent again, that's not incorrect or a bad practice, that's exactly the way you should be doing that ;)

Champion

I see, thanks for your reply! I was just wondering if I could be doing it better haha.

Really nice article! My congrats...

The only thing I don't really like about light BPT (it is indeed present on regular BPT also) is the item #4 in your list, the one related to unhandled exceptions.

If you have lots of transactions failing due to a software malfunction (that will only be fixed on a next version), it can easily turn into a logging nightmare.

João Melo wrote:

Really nice article! My congrats...

The only thing I don't really like about light BPT (it is indeed present on regular BPT also) is the item #4 in your list, the one related to unhandled exceptions.

If you have lots of transactions failing due to a software malfunction (that will only be fixed on a next version), it can easily turn into a logging nightmare.

Hi João,


Thank you very much :)


I understand your concern. If this indeed became a problem, probably I would create a business error queue for all the light BPTs that generated errors, then I could be handling them manually.

On the light BPT processes I would add an Exception Handler catching all exceptions, and simply add it to the business error queue together with the error message to be able to track what cause it to fail in the first place.

This way if there was an error, the process would simply end because we handled the exception.

On the business error queues created, we could force a new light bpt to start by creating a new record on the business event table.


This would solve your problem and you could manage it the way you want.


Cláudio Oliveira wrote:

João Melo wrote:

Really nice article! My congrats...

The only thing I don't really like about light BPT (it is indeed present on regular BPT also) is the item #4 in your list, the one related to unhandled exceptions.

If you have lots of transactions failing due to a software malfunction (that will only be fixed on a next version), it can easily turn into a logging nightmare.

Hi João,


Thank you very much :)


I understand your concern. If this indeed became a problem, probably I would create a business error queue for all the light BPTs that generated errors, then I could be handling them manually.

On the light BPT processes I would add an Exception Handler catching all exceptions, and simply add it to the business error queue together with the error message to be able to track what cause it to fail in the first place.

This way if there was an error, the process would simply end because we handled the exception.

On the business error queues created, we could force a new light bpt to start by creating a new record on the business event table.


This would solve your problem and you could manage it the way you want.



That can work. But we're creating a business concept / process to handle tech exceptions... :/ Sounds more like a workaround than a proper solution.

João Melo wrote:

Cláudio Oliveira wrote:

João Melo wrote:

Really nice article! My congrats...

The only thing I don't really like about light BPT (it is indeed present on regular BPT also) is the item #4 in your list, the one related to unhandled exceptions.

If you have lots of transactions failing due to a software malfunction (that will only be fixed on a next version), it can easily turn into a logging nightmare.

Hi João,


Thank you very much :)


I understand your concern. If this indeed became a problem, probably I would create a business error queue for all the light BPTs that generated errors, then I could be handling them manually.

On the light BPT processes I would add an Exception Handler catching all exceptions, and simply add it to the business error queue together with the error message to be able to track what cause it to fail in the first place.

This way if there was an error, the process would simply end because we handled the exception.

On the business error queues created, we could force a new light bpt to start by creating a new record on the business event table.


This would solve your problem and you could manage it the way you want.



That can work. But we're creating a business concept / process to handle tech exceptions... :/ Sounds more like a workaround than a proper solution.

I can understand why Outsystems chose to follow this direction with Light BPT, if there was an unhandled exception and the transaction would simply stop without postponing, we would be left with no trace of that transaction and wouldn't have knowledge which ones were unsuccessful.


On a project I'm in, we use it to handle pub/sub webservice messages between systems, and I created a retry mechanism that if it fails, it retries X number of times, if it fails those X times, it goes to a Manual Intervention queue with a reason it failed, then we can create any kind of alerts to be notified on this and take a manual action.


Hope this can be a solution for your specific business case.

it is nice if only there is the Java Stack version. I am doing heavy BPT for banking workflow and it is Java.

Optimus Prime wrote:

it is nice if only there is the Java Stack version. I am doing heavy BPT for banking workflow and it is Java.

Hi Optimus Prime,

What version are you using? Light BPT is only available on OS10.


João Melo wrote:

Optimus Prime wrote:

it is nice if only there is the Java Stack version. I am doing heavy BPT for banking workflow and it is Java.

Hi Optimus Prime,

What version are you using? Light BPT is only available on OS10.



I mean there is no Java Stack for Light BPT

Hi Cláudio!


"Expose Process Events" option tells that "creating or updating entity records issues events to process", but in "Launch On" process option I can see only Create<Entity> as you used in your example; there is no Update<Entity> availabe.


There are some step to enable it? 


Is it possible to use CreateOrUpdate<Entity> event?


I have a process that I need to start execution when insert or update occur in an entity.


Thanks,


Tiago


Cláudio Oliveira wrote:

Hello developers!

 

While messing around with Light Processes, I've realized that there is a lack of information on how it works, so I've decided to give some tips.


You can already find some information on how to activate and on which conditions a BPT Process will be triggered as a Light Process:

But there is more relevant information that in my opinion we all should know.


Useful information / tips:

  1. Light Processes will skip the execution of callback actions (like "On Process Start")

  2. Light Processes won't allow you to add an Output parameter to the Process (if you do, you'll have an internal error on the compiling step while deploying)

  3. By default, Light Processes have 20 threads per front-end, this threads are separate from BPT threads (normal BPT have 10 threads by default).

  4. If an unhandled exception occurs during the execution of a Light Process, the Scheduler will pick that Light Process again moments later (the number of times it failed to execute will increase how much in the future that process will be picked again, until it reaches a point that it will be picked once a day)

  5. Light Processes are executed on its own transaction (session variables and GetUser() will be empty unless you define them during its execution)

  6. Light Processes execution is asynchronous (the transaction responsible for triggering the Light Process will not be waiting for the execution of the process, the process will run on the background)


When to use?

  • Light processes are intended to be used for background processing

  • Replace timers that don’t require scheduling by Light Processes

  • Replace normal BPT Processes only used for background processing containing only 1 automatic activity

    • Even if the normal BPT Process has more than 1 automatic activity, it might be the case that the main workflow (for the 80% of the cases) could be made with a Light Process and only on the exception scenarios

    • For the exception scenarios a normal BPT would be launched via “LaunchProcess” action

    • This could alleviate the server load on the normal BPT Process threads and use the more performant 20 threads from the Light BPT

    • Light Processes don’t create entries on BPT tables, which means less entries on tables Activity, Activity_Output, Process, Process_Input and Process_Output


How does Light Process actually works under the hood?

  1. When defining an Entity “Expose Process Events = True”, the deployment will create an SQL event table containing information relevant for the following triggers

    • The trigger (mentioned further) will look at this table to know which ProcessDefinition will be used and if is Light Process or normal BPT process


  2. After publish, the deployment step will create an SQL table to handle the events over the “File_Event” user table


  3. Create a BPT Process and define the “Launch On” to “Create <your entity>”

    • When you do this, during the deployment, a new record will be added to the event table OSEVT_xcd_File_Event mentioned under the point 2

      1. _PROCESS_DEF_ID = 52 corresponds to the process Background_ProcessFile

      2. _IS_LIGHT is set to 1 because this BPT Process matches all the conditions to be a Light Process

  4. A Trigger will also be created for the normal user table (in this case for the File_Event)

  5. When a record is inserted in the user table, the trigger will:

    • Query the event table OSEVT_xcd_File_Event to fetch the _PROCESS_DEF_ID

    • Insert on the system SQL table ossys_BPM_Event (next point will explain this table) the fetched information together with the Entity_Record_Id which identify the Primary Key from the user table File_Event



  6. Scheduler is constantly pinging the system table ossys_BPM_Event for a record with Next_Run <= Current Date, once found, it allocates a thread to handle that event

    • The running thread mentions the Event id 10 which corresponds to the record being picked from the ossys_BPM_Event

  7. In case an Unhandled Exception occurs during the execution, the table column ossys_BPM_Event.Error_Count will increase by 1 and the ossys_BPM_Event.Next_Run will be set to moments later

    • The number of times it failed to execute will increase how much in the future the Next_Run will be set, until it reaches a point that it will be set to be picked once a day



Hi Tiago,


Indeed only the Create<Entity> is available and there is no way to enable the Update<Entity>, but this doesn't mean you can't do what you want.


With Light BPT, I would recommend to create an event entity only to handle the trigger of new instances.

  1. First you should create wrappers for the methods Create, Update and CreateOrUpdate of your business entity that is now being used to trigger Light Processes.
  2. Create an Event entity for your business entity "Business_Event" (Id PK, BusinessId FK).
  3. On the wrappers you created for your business entity, always create a record on your "Business_Event"
  4. Change your Light Process Create<Entity> to use the new Event entity instead.


So now the entity used to trigger the Light Processes is not the Business entity, but the Event entity.

Hope this help you.

Tiago de Resende wrote:

Hi Cláudio!


"Expose Process Events" option tells that "creating or updating entity records issues events to process", but in "Launch On" process option I can see only Create<Entity> as you used in your example; there is no Update<Entity> availabe.


There are some step to enable it? 


Is it possible to use CreateOrUpdate<Entity> event?


I have a process that I need to start execution when insert or update occur in an entity.


Thanks,


Tiago


Cláudio Oliveira wrote:

Hello developers!

 

While messing around with Light Processes, I've realized that there is a lack of information on how it works, so I've decided to give some tips.


You can already find some information on how to activate and on which conditions a BPT Process will be triggered as a Light Process:

But there is more relevant information that in my opinion we all should know.


Useful information / tips:

  1. Light Processes will skip the execution of callback actions (like "On Process Start")

  2. Light Processes won't allow you to add an Output parameter to the Process (if you do, you'll have an internal error on the compiling step while deploying)

  3. By default, Light Processes have 20 threads per front-end, this threads are separate from BPT threads (normal BPT have 10 threads by default).

  4. If an unhandled exception occurs during the execution of a Light Process, the Scheduler will pick that Light Process again moments later (the number of times it failed to execute will increase how much in the future that process will be picked again, until it reaches a point that it will be picked once a day)

  5. Light Processes are executed on its own transaction (session variables and GetUser() will be empty unless you define them during its execution)

  6. Light Processes execution is asynchronous (the transaction responsible for triggering the Light Process will not be waiting for the execution of the process, the process will run on the background)


When to use?

  • Light processes are intended to be used for background processing

  • Replace timers that don’t require scheduling by Light Processes

  • Replace normal BPT Processes only used for background processing containing only 1 automatic activity

    • Even if the normal BPT Process has more than 1 automatic activity, it might be the case that the main workflow (for the 80% of the cases) could be made with a Light Process and only on the exception scenarios

    • For the exception scenarios a normal BPT would be launched via “LaunchProcess” action

    • This could alleviate the server load on the normal BPT Process threads and use the more performant 20 threads from the Light BPT

    • Light Processes don’t create entries on BPT tables, which means less entries on tables Activity, Activity_Output, Process, Process_Input and Process_Output


How does Light Process actually works under the hood?

  1. When defining an Entity “Expose Process Events = True”, the deployment will create an SQL event table containing information relevant for the following triggers

    • The trigger (mentioned further) will look at this table to know which ProcessDefinition will be used and if is Light Process or normal BPT process


  2. After publish, the deployment step will create an SQL table to handle the events over the “File_Event” user table


  3. Create a BPT Process and define the “Launch On” to “Create <your entity>”

    • When you do this, during the deployment, a new record will be added to the event table OSEVT_xcd_File_Event mentioned under the point 2

      1. _PROCESS_DEF_ID = 52 corresponds to the process Background_ProcessFile

      2. _IS_LIGHT is set to 1 because this BPT Process matches all the conditions to be a Light Process

  4. A Trigger will also be created for the normal user table (in this case for the File_Event)

  5. When a record is inserted in the user table, the trigger will:

    • Query the event table OSEVT_xcd_File_Event to fetch the _PROCESS_DEF_ID

    • Insert on the system SQL table ossys_BPM_Event (next point will explain this table) the fetched information together with the Entity_Record_Id which identify the Primary Key from the user table File_Event



  6. Scheduler is constantly pinging the system table ossys_BPM_Event for a record with Next_Run <= Current Date, once found, it allocates a thread to handle that event

    • The running thread mentions the Event id 10 which corresponds to the record being picked from the ossys_BPM_Event

  7. In case an Unhandled Exception occurs during the execution, the table column ossys_BPM_Event.Error_Count will increase by 1 and the ossys_BPM_Event.Next_Run will be set to moments later

    • The number of times it failed to execute will increase how much in the future the Next_Run will be set, until it reaches a point that it will be set to be picked once a day






Hi Claudio,

thank you for your reply; we can implement this without a lot of customization, indeed we already use wrappers for business entities.

Actually I'm a little disappointed because there is no way to get update event directly from the Business Entity; just reading "Expose Process Events" hint, I thought that platform will handle update events by default...


Regards,

Tiago


Cláudio Oliveira wrote:

Hi Tiago,


Indeed only the Create<Entity> is available and there is no way to enable the Update<Entity>, but this doesn't mean you can't do what you want.


With Light BPT, I would recommend to create an event entity only to handle the trigger of new instances.

  1. First you should create wrappers for the methods Create, Update and CreateOrUpdate of your business entity that is now being used to trigger Light Processes.
  2. Create an Event entity for your business entity "Business_Event" (Id PK, BusinessId FK).
  3. On the wrappers you created for your business entity, always create a record on your "Business_Event"
  4. Change your Light Process Create<Entity> to use the new Event entity instead.


So now the entity used to trigger the Light Processes is not the Business entity, but the Event entity.

Hope this help you.

Hi Claudio,


Is any specific licensing is needed besides the Business Process Execution and Modeling?


The checkbox to activate light processes is disabled in our case (version 10.0.1010.0)....


Regards,


Artur

Artur Resende wrote:

Hi Claudio,


Is any specific licensing is needed besides the Business Process Execution and Modeling?


The checkbox to activate light processes is disabled in our case (version 10.0.1010.0)....


Regards,


Artur


Hi Artur,


I'm not sure if on version 10, Light BPT are readily available. 

If I'm not mistaken, you have to request Outsystems to activate it via Support Portal, but I can be wrong :/


Still if you request via Support Portal, Outsystems should help you setting it up.


Kind regards,

Cláudio Oliveira

Thank you Cláudio!


I will try the Support Portal.


Regards


Artur 

Hi Artur,

In case you did not yet have the answer, and for those looking for the answer to why Light BPT is not avaiable:

It was introduced in Platform server 10.0.804.0, see https://success.outsystems.com/Support/Release_Notes/10/Platform_Server


So in your case you need to update the platform server

Hope this helps.


Shahin


Artur Resende wrote:

Thank you Cláudio!


I will try the Support Portal.


Regards


Artur 



Shahin Keshavari wrote:

Hi Artur,

In case you did not yet have the answer, and for those looking for the answer to why Light BPT is not avaiable:

It was introduced in Platform server 10.0.804.0, see https://success.outsystems.com/Support/Release_Notes/10/Platform_Server


So in your case you need to update the platform server

Hope this helps.


Shahin


Artur Resende wrote:

Thank you Cláudio!


I will try the Support Portal.


Regards


Artur 



Hi Shahim,


Thank you for the help.


We are using release 10.0.1010.0 that is more recent, and the option is shown on Service Studio, but has the Checkbox disabled, invalidating the possibility to activate it.

We opened a case in the Support Portal and are waiting for a reply.


Regards

Artur


Hello guys!

When we use a Light BPT attached in LauncheOn event it's running fine, but if in the same espace we try to launcher this event manually is it supposed that they run like a light BPT or BPT instead ?


Thiago Galvão Soares wrote:

Hello guys!

When we use a Light BPT attached in LauncheOn event it's running fine, but if in the same espace we try to launcher this event manually is it supposed that they run like a light BPT or BPT instead ?


 

 Hi Thiago,


One of the requirements for lightprocesses is that they are launched with an Create event (Database trigger event). So when you use the LaunchProcess, it won't be a light process.

Hello Shahin,


If you launch the same Process via the LaunchProcess it will run as a normal BPT Process, creating entries on the ossys_bpm_process and ossys_bpm_activity entities (which does not happen on the LightBPT).

Is there a way to increase the number of threads for LightBPT processes?
I understand it's set to 20 right now and it seems to be a "default" setting.
(Sounds to me like an IIS limit set)

We have a dedicated server set to deal with Kafka events and we see that the events are being throttled (perhaps the 20 threads?) and thus would like to increase the number of threads.
The server is running idle and we'd like to make it do some work.

My NGINX experience has taught me that is is really worth to tune a webserver for it's purpose, which is our architectural intent with this one.

I've found this page on IIS threads:
https://learn.microsoft.com/en-us/office/client-developer/access/desktop-database-reference/specifying-threads-per-processor-on-iis

Sounds to me like something to try but I rather get a response from anyone that -really- knows the in's and out's on this so the approach is less "Cowboy".

Hi Eric,

As Cláudio mentioned, I think it is indeed configurable with the Configuration Tool in on-prem environments, or manually from IIS.

However it's worth to be careful with this approach; having more than 20 (more than suggested) simultaneous threads might cause a bottleneck somewhere else, at DB touch points for example.

Hi Ozan,

Thank you for your response.

It doesn’t seem to be in the configuration tool.

I am aware of the bottlenecks, thanks for bringing this up though.

I currently can’t really analyze what is going on given the “throttling” of the static threads so I am looking for the actual resource huggers (if there are any)

One of the downsides of the TCP protocol is the time for setting up a connection. This in conjunction with the thread limit can work as a true speed limiter hence my focus on this too.

Hey Eric,


Long time ago, I remember having a talk with OutSystems about this topic and that if we wanted we could request to make that change. (Since we are on PaaS we don't have direct access to our infrastructure so OutSystems Support would be the ones to make this change).


Since we never ended up doing it, I'm not 100% sure it's possible, but I think it is.


If you are on Premise or Private Cloud, I would suggest o check the "Configuration Tool", there you can change the number of Threads for Timers, maybe there's also an option for BPT and Light Processes.


If not, I would recommend opening a Support Case making the request to OutSystems.


If you get a definitive answer and are able to do it, could you share here the outcome?


Kind regards,

Cláudio Oliveira

Thanks for your response, Cláudio.
I will share the OutCome ;) if I get a definitive answer.

The hyperlink of the article above is not valid anymore.  

Here is the valid hyperlink to OutSystems 11 documentation:

Design Scalable Database Queueing Using Light Processes

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.