Resilience. It is such a powerful word, and one of the most sought-after outcomes for a software developer when building an application.
In this article, we will take a look at how you can use a well-known pattern from software architecture — Publish-Subscribe (Pub/Sub) — to create resilient, decoupled, and reliable applications. We’ll do it using asynchronous capabilities from the OutSystems Platform in combination with Amazon’s Simple Notification Service (SNS).
Why do apps need to be resilient? Building enterprise systems that integrate with external tools can be a painful mission in software engineering, as your application depends on non-functional requirements that are out of your hands.
The same applies to large applications, which traditionally are distributed systems or microservices running across the network where you also don’t control the entire ecosystem.
In this scenario, your application risks losing responsiveness or becoming unavailable to users. To counter this risk, you must make the app resilient to failure and ensure it will work properly even under poor conditions. Thanks to the previous generations of software engineers, such issues have already been handled.
How? One way they successfully addressed these was using the Pub/Sub pattern. This solution has proven benefits in decoupling modules or apps, preparing them to work asynchronously, and consequently, handle peaks of usage, which means they can weather and be available under unfavorable conditions.
This pattern is particularly interesting when an app generates information that could be useful to many other modules or applications.
This is a short yet accurate definition from Wikipedia:
“Publish-Subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.”
Using Amazon’s SNS
Amazon’s Simple Notification Service is a fully-managed messaging service for both application-to-application (A2A) and application-to-person (A2P). Here we will focus on the A2A scenario. To show how this works, we will create a module to publish messages on an SNS topic and two other modules to receive messages from an SNS topic; these modules will also use the asynchronous Light BPT feature from OutSystems for added reliability.
To follow this guide, you need to:
- Have access to an OutSystems development environment (the free edition is enough).
- Have basic knowledge of OutSystems development and tools such as Service Studio IDE, Service Center, and Forge.
- Have an account on Amazon Web Service or create a new one (we'll use free tier tools).
The plan is to create a very simple User Management application in which an administrator can create other users, providing basic information about them. These data will be used on the other two applications that will receive, store, and process it in a particular way. We will not do it directly: the first application will post data into an SNS topic, and the SNS will deliver it to the subscribers through an HTTPS endpoint. SNS provides other protocols to fan out messages. You can take a look here if you want to learn more.
With that said, let's start! It’s hands-on time!
Creating the Application
We will create an OutSystems application within three modules — Publisher, Subscriber 1, and Subscriber 2. Note that this is not important for the entire solution since each module could be built in any environment or technology. You could have only the publisher as an OutSystems module and subscribers as external ones, or you could have everything on the OutSystems side, but with each module in a different application. It entirely depends on your use case, and we will use one application just for brevity.
Go to Service Studio, create a new Reactive Application with one Reactive module, and publish it. I named it Pub/Sub Application and PublisherApplication, respectively.
Our first module will represent the publisher application and has three goals:
- Handle the user input through an interface.
- Store the created items by the users on the database.
- Send the created items to the SNS topic.
We will focus on these few requirements to achieve our goal to show the basics of using Pub/Sub with OutSystems and SNS. However, it is common to see publisher applications also dealing with topic operations like lists and creation. For now, we'll perform these admin activities directly on AWS Console.
On the created module, add an Entity called ApplicationUser and new attributes; Name (Text Data type), Email (Email Data type), and SNSMessageId (Text Data type):
Now, make use of the OutSystems scaffolding to create the screens. Go to the interface module, double click MainFlow, and drag the ApplicationUser entity into the flow area. Two screens will appear: ApplicationUsers and ApplicationUserDetail. Publish your work, and with these simple steps, your module should achieve its first and second goals. If you are not familiar with the scaffolding usage, take a break to navigate the module to see what was created for you.
We need to work on the third goal now and send the created records to SNS. To do this, we will use the SNS Connector component from OutSystems Forge. This component is based on AWS SDK for .NET and handles communication with SNS, allowing you to focus on what’s important — your business logic — instead of investing your efforts in developing the requirement. Download it and install it through Service Studio.
Creating the AWS SNS Topic
It's time to take a break on the OutSystems side and create the AWS SNS topic. You can do that following these few instructions from AWS:
- Setting up access for Amazon SNS — follow steps 1 and 2.
- Getting started with Amazon SNS — follow only step 1.
Take note of some of the assets created on these steps, such as the region where you made the topic, the topic ARN, the Access and Secret Keys for the created user — we will need them soon.
Coming back to the Publisher module, open the dependency tool and filter by AmazonSNS; you should see some actions to be used from the installed Forge component:
For now, we will use only the Topic_Publish public action. Select it and apply the dependency.
At this point, we are ready to send messages to SNS, right? We could just modify the ApplicationUserCreateOrUpdate server action (where an insert occurs in the database), including a call to Topic_Publish from Amazon SNS Connector in the flow. The message would be sent just after (or before) its creation in the local database.
OutSystems Light BPT to the Rescue
But what happens if we have an issue like a firewall or proxy blocking our connection to SNS? Or the operation takes too long due to network problems?
Then, OutSystems Light BPT comes to the rescue. We can use it to trigger an event and send the messages asynchronously, protecting our application from losing data or responsiveness if an external problem happens.
Light BPT is an excellent option to trigger asynchronous actions based on events that occur in entities. When using it, your process must contain only one automatic activity. And because of that, it consumes less infrastructure when compared to a typical process on the OutSystems Platform; it also offers automatic retries if an error occurs during its execution.
To use it, go back to the ApplicationUser entity on the Data tab, select Edit Entity, and mark Expose Process Events on the Advanced tab. This way, we can use the creation event in this entity as a trigger for our logic process:
One more step is required to use Light BPT: access the Service Center, search the module you are working on, and set the property Light process execution on the Operation tab.
With this done, go back to Service Studio, go to the Processes tab, add a new process, and set the Launch On property to CreateApplicationUser function. This will trigger the process execution when a new record is inserted on the ApplicationUser entity.
On the process flow, drag the Automatic Activity widget. Note that there is an ApplicationUserId input parameter available that we will use in the next step. Publish your module, and you should see a message: "Process Compilation: Process 'SendMessageToSNS' is using light process execution. Process history will not be recorded." It certifies that you are using a light version of BPT technology.
Completing the Logic Flow
Now, let's proceed and create our processing logic to send records to SNS.
Double click on the Automatic Activity you've created previously. It will open a new empty flow, just like a simple server action. Here we will:
- Get Application User details based on the ID received by the trigger.
- Serialize the content in a JSON format.
- Set up our credentials and region to publish on AWS SNS.
- Send the message using the Topic_Publish action from the SNS Connector.
- And finally, get the ID returned by SNS and update the ApplicationUser entity record in our database.
Your complete flow should look like this:
This flow is just an example; in your application it must represent your needs to ensure you can do everything you want before or after sending a message.
An important note about SNS Connector is that it uses SDK for .NET v3, and it internally creates the endpoint to SNS based on the region provided in one parameter. But the point is that the regions inside this class have different codes from the AWS Console. So, you need to check the region code on this link to provide the correct input parameter and avoid invalid region errors. I'm using the South America (São Paulo) region, which appears as sa-east-1 on the AWS Console and SAEast1 in the internal SDK class.
All set from the Pub perspective. You can test your application by creating a new user and keeping the SNS Message field empty.
After clicking Save, you will be redirected to the list where the SNS Message is still blank. Go to the detail screen or refresh the list screen, and the SNS Message field should be updated. This happens because when you save the record in the application, it is kept in the local database, triggering the process that updates this attribute on the database after sending the content to SNS.
At this point, we are creating records locally and sending them to Amazon SNS. Let's now go to the other side, to the modules that will receive these messages.
Creating the Subscribers
Let's create the first subscriber — this module has different objectives:
- Subscribe and unsubscribe to topics created on SNS.
- Expose an HTTPS endpoint to receive messages from AWS.
- Handle subscription and unsubscription confirmation requests.
- Store messages received from SNS.
With this in mind, let's create a new module to act as a Subscriber application:
Here we will create an Entity to hold received messages and a screen to see them. On the Data tab, create an entity named ReceivedMessage and add a text attribute called ReceivedMessage. Then, change its Length property to 500. Move to the Interface tab, open the MainFlow, and drag and drop the created Entity there. As expected, it will create two screens — ReceivedMessages and ReceivedMessageDetail.
We can delete the ReceivedMessageDetail screen because we just want to see the received messages in this module, which can be done on the list screen. By doing so, we will get two errors due to the redirects to the deleted screen. For the first one, we can just delete the link widget from the table widget. The second is in the button used to add new messages. Change the text from Add Received Message to Subscribe, and select New Client Action for the OnClick event. You can remove the other unused elements to avoid warnings in your module.
Building REST API
The client action you created will be used to subscribe to a topic on AWS SNS and then start receiving messages. But to subscribe to a topic, we must provide Protocol and Endpoint parameters; AWS uses these values to deliver messages that arrive on the topic to the destination. In our case, we will use HTTPS as the protocol and a REST API method as the endpoint, but first, we need to build it.
To create the REST API, move to the Logic tab, right-click on REST, under Integrations, and select Expose REST API. Name it MessageAPI, right-click on it and choose Add REST API Method. Name the created item Add and change the HTTP Method property to POST.
Notice that there is a URL property that points to your created API Method. This will be used as the endpoint when subscribing to a topic. The complete URL is formed by https://<environment-adress>
Before returning to the action that will subscribe to the topic, we must develop the processing logic responsible for confirming subscription and unsubscription. Why? After sending a subscription request to AWS, it will send a confirmation message to the endpoint to assert that you want to subscribe or unsubscribe to that topic, so your endpoint should be able to handle confirmation messages.
To do this, get Subscription_Confirm, Subscription_Unsubscribe, and Topic_Subscribe actions from AmazonSNS in the Manage Dependencies area and apply these new dependencies to our module.
To handle these messages, we need to know what kind of information and structures we will receive on this endpoint. Thankfully, AWS has great documentation which shows us how they do it:
POST / HTTP/1.1
Content-Type: text/plain; charset=UTF-8
User-Agent: Amazon Simple Notification Service Agent
"Type" : "SubscriptionConfirmation",
"MessageId" : "165545c9-2a5c-472c-8df2-7ff2be2b3b1b",
"Token" : "2336412f37f...",
"TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
"Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-west-2:123456789012:MyTopic.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
"SubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-west-2:123456789012:MyTopic&Token=2336412f37...",
"Timestamp" : "2012-04-26T20:45:04.751Z",
"SignatureVersion" : "1",
"Signature" : "EXAMPLEpH+...",
"SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem"
Driving API Development
We can extract some relevant pieces from this example that will drive our API development. AWS sends the information by doing an
HTTP POST. In the headers area, the Content-Type is sent as text/plain; and in the body area, we should get all the information we need for future use. At this moment, however, the most important attribute is Type.
We had configured our API method to receive requests through
POST. To receive body information sent as text/plain by AWS, we need to create one input parameter with Data Type
Text and a Receive In property as
Body in the Add API method. Instead of receiving all body data in one text parameter, we usually create one structure representing body structure.
The platform automatically performs deserialization for us, but in this case, that is unfortunately not possible since the sent media type is text/plain, which is not accepted in OutSystems APIs when the input parameter is a Record or List (see here for more information). In our case, we will still create a structure and use it as a local variable to hold the received body data.
On the flow, we can now deserialize the received values in JSON format into the local variable and use its values to verify the type of message.
Using the If widget, we can check the message type and call the Subscription_Confirm action if it’s equal to SubscriptionConfirmation. You need to set the AWS Credentials (you can use the ones you defined for this article’s purposes, but for real-world use, there are a set of best practices to follow) and Token / TopicArn parameters using the information from the Body local variable.
With all set for confirmation, we can go back and finish the Subscribe process. Create a new server action on the SubscribeOnClick client action. In this server action, you need to set the AWS Credentials (again, observe the best practices for real-life applications) and call the Topic_Subscribe action.
You need to fill in the protocol (HTTPS), the endpoint (the complete URL for the previously created REST API method), and the TopicArn from AWS.
Call this server action on the SubscribeOnClick client action and add a message widget to provide visual feedback. Then, publish your work, open the application, and click on the Subscribe button. Besides getting a feedback message from the screen, you should see a new Topic subscription in the AWS Console. If it all works, you'll see a record confirmation like this:
If you subscribe before building the confirmation flow or some error occurred in the confirmation process, you'll see a pending item on the list. In this case, you can check Service Center logs to recognize what is wrong and then send another request confirmation from AWS Console.
Now, we just need to adapt our API flow to register the received message when it is a notification from another application.
Registering a Message From Another App
Actually, this is the easiest part: go back to the Add Method API and add a Local variable with ReceivedMessage data type. Assign the Body.Message value to the Message attribute of this variable and call the CreateReivedMessage action.
To finally test your hard work, you can open the Publisher module and create a new application user. On this side, nothing will change, after saving it and waiting a bit, it should show the SNS Message Id as a signal that its work is done.
On the Subscriber side, you should open the application and see a new record on the list with a JSON as the received message content. If something didn’t work properly, use Service Center to check the extension calls for the SNS Connector and the integration call for the exposed REST API, both under the Monitoring tab.
We could now create another Subscriber to see the whole thing in action. A simple way to do so is by cloning the subscriber exiting module and changing the endpoint value when subscribing. Open the Subscriber module in Service Studio and select Clone, which is available in the Module menu.
The above message will appear, and a new module will open in Service Studio. Change the module name under module properties if you’d like (I'm using SubsbriberApplicationTwo) and change the Endpoint parameter where we call the Topic_Subscribe for the value of the new module you created (in my case /SubscriberApplicationTwo/ instead of /SubscribeApplication/, which is the first completed module).
You should now publish your cloned module, access it, and subscribe to the topic using the Subscribe button. Later, you should move your cloned module from the Independent Modules application to your Pub/Sub Application.
Well done! From now on, every record created on the Publisher application will be delivered to both subscribers! We also achieved our main goal: to keep the Publisher application decoupled from its subscribers and safe if one of them becomes slower or unavailable.
In our example, both Subscribers receive and store the messages, but in a more realistic scenario, each of them could build the flow and implement it whenever necessary with the received data. A good approach here is also using Light BPT to do the job after receiving the messages and preparing it to deal with external dependencies or long activities that may happen asynchronously.
You may have noticed that we did not implement the Unsubscribe flow. You can try to do it by yourself using the Subscription_Unsubscribe action from SNS Connector. After unsubscribing, the application should stop receiving new messages from the Publisher.
In a future article, we will cover security aspects for this use case. I made some notes about AWS’s best practices involving security, and there are other vital improvements to be addressed that I will be sharing. Watch this space!