Q: Expose Integration API - Best practise

Hi,

I'm looking for some advice regarding exposing an API that will receive integration messages from our service bus for example.

Use case: 

I have a service bus that will send me a message (REST - POST) whenever some data changed in a source system. This message will contain a multilayered json that I will need to parse before I can store the data in the database. This is because I need to perform mappings with other database entities before I can store the changes.
Example; in the json is the internal code of a project but this code is not the primary key in our OutSystems database. So I first need to find the record and attach the record ID to the entity record that I want to store. 

Since a lot data can change at once I'm trying to find out what the best practice would be for handling this data volume. I can think of 2 possibilities. 


Option 1:

Do nothing special. Simply handle the message whenever it's received by OutSystems.
Pro: Easy understandable readable flow for any developer. Easy to debug in case of issues.
Con: 

  • It will lock up resources and will probably cause HTTP 408 errors to be returned to our service bus.
  • This will also have impact on other applications that are running on the same server since all CPU or Disk or Memory resources are being locked.

Question: How many concurrent API requests can OutSystems handle? Is there a set limit or is it based on the systems capabilities?


Option 2:

Use BPT. When the message is received, convert the data to a json string and store it in the database. Then have a BPT process trigger on creation of this record and parse the data in that separate process. After processing, delete the record
Pro: It allows the systems to collect the messages as quickly as possible and close the session with the service bus quickly. The data will then be processed async (This is ok for these kind of integrations). As far as I understand you can only run so much BPT processes simultaneously and this will probably prevent an overload of our system
Con

  • Debugging becomes a lot harder. 
  • It's more complex solution that requires a bit more documentation and hand-over. 
  • The service bus will only get 200 - OK messages and will not be notified if the data is invalid for whatever reason. 


How do you guys handle these kind of processes? And for what option would you choose? Or would you do it entirely different? I would love to hear your suggestions and feedback!


Hi Vincent,

Both solutions are valid, but it's going to depend on how much data and how often you'll be receiving it. I can tell you that I implemented option 1 on a project a couple of months ago, and my use case was very similar to yours (complex XML would be sent to a web service on a daily basis, but a limited number of times). But the data was controlled and there was very little risk of timeouts.

Option 2 is more adequate if you're going to receive larger amounts of data. Coincidentally, the last Meetup had an Outsystems architect presenting a similar use case, explaining how you can use Light BPTs to implement this in a scalable way. I've spoken with the Meetup organizer and the video will be released today, I'll link it as soon as its up - I think it will be very useful to illustrate this.

There's also an option 3: processing in bulk via a Timer.

As for the con in your option 2: if you need so much time processing the message that you can't use option 1, there's just no way you can respond in time anyway, so I wouldn't call it a con of that option per se.

Here are the Meetup materials I was referring to:

https://www.dropbox.com/sh/idvwkgwq7k2zpqk/AAASx60PNEvmijrA6zejcZmPa?dl=0

Check the LBPT_Presentation.pdf and the mp4 recording and let us know if they were of any help.

@Kilian: Bulk will be certainly be used for the initial import of the data. But for most of our integrations we want a near real-time update of the data.

@Afonso: Thanks for the feedback and the material. I'll take a look at it :)

Hi Vincent

As you say a lot of data can change at once, believe concurrency is a key requirement for you. 

Light Process may be an option, as you get a high level of parallel processing, if not real time, also you can use Timers for your initial import (As you already mentioned). 

You can enable Light Process Execution from service center. 

Hope it helps !!!