Hi All,

I need some suggestions regarding the continuous monitoring process. Please see the details below.

CASE :

I have two queues (SQS) configured in an AWS environment that is used by the client automated process to send requests and fetch process requests from it. I have a logic that gets the requests from the first queue validate- process and save it and then sends the updated result back to the second queue. I have already designed a working logic that does the same, but is need to know is there any way to constantly monitor the queue?

SOLUTION TRIED/PROPOSED :

1. For now, I am using a timer to monitor the SQS that does the same, but the catch here is, the requests that are put in the queue take time to process as it is executed when the timer runs and maximum timer time difference can not be more than 10 mins because of platform restrictions (there are multiple timer processes so to decrease the load the timer run has been configured to execute in proper time).

2. The second approach is exposing an API, but I doubt about the performance here. The client application will be calling the API whenever it receives request in the Queue. There is a situation where multiple requests can arise at the same time. As per what I got the info, client application is capable of handling multiple requests at a time, that is it can sent multiple API requests at a time but will the outsystems application be capable to handle so many requests.

Any suggestions will be appreciated.

Thank you :)




Solution

Hi Pranav,

I had a similar architecture with a Send and Receive AWS SQS queue. We configured the queues with long polling (which has a max of 20seconds) on the WaitMessage. Then we also received a max of 10 messages on each call. Still used a timer that would restart itself continuously, for the customer's use case this was sufficient.

I did look into alternatives, and the most interesting one is to use AWS SQS Triggers, to start an OutSystems REST API. Your OutSystems Infrastructure can handle multiple requests, depending on the volume you may need more front-end servers.

The reason we didn't implement this is that we were required to use FIFO queues and that is not supported with SQS Triggers. But if you don't have FIFO queues, I should check out the SQS triggers. Be aware that calling SQS triggers are not free and are charged by AWS.

Regards,

Daniel

Solution

Hi Daniel,

Thanks for the reply.

yes, I just got that information that triggering API from the queue will be costly and hence the will not be allowed by our application architect. 

Currently, I am getting a single queue message at a go. As per what you did, I am also planning to increase the message count to 10 for each request. This will help me tackle the situation a bit. 

Seems there is no other alternative.

Thank you for your help, really helpful.

:)