At OutSystems, we often get asked how low-code rapid application development platforms can scale. We know low-code scales well, because we have hundreds of customers running applications on our platform that serve millions of users. But, we decided to demonstrate just how scalable a low-code platform can be. So, we set up a benchmark to test our low-code platform under varying levels of load from simulated users.
Our goals for this project were to:
- Demonstrate the scalability that is achievable with our platform using a mobile banking application as an example.
- Simulate the scale that would be needed by a very large bank running this mobile banking application at peak user load times.
- Base our experiment on real customer performance data from a production mobile banking application. Then extrapolate from that data to set our target performance goals under a massive load situation for our simulated application.
An application or system is said to be scalable if, by adding more hardware, the application can linearly handle more requests than before. Another way to put it is that a system is said to be scalable if its performance doesn’t degrade significantly as the number of users, or equivalently, the load on the system increases.
In a non-scalable system, as user and transaction load is increased, it will reach a point at which all system compute resources are consumed. The response time will degrade and eventually the system will fail.
In a scalable system, as the user and transaction volumes increase additional compute resources are added and the application response time remains constant with the increasing load on the system. This is the scenario that will keep all users happy regardless of the number of users on the system.
Our Test Scenario
One of our customers, a regional bank, has 1.4 million customers. We learned that they typically have 83,718 active mobile banking users during a peak time. They also shared that during this peak time, their systems saw an average transaction rate, or TPS, of 12.2 transactions per second. They experience a maximum transaction rate of 47 transactions per second during a peak time.
Based on publicly available data from one of the largest banks in the US, we learned that they can have 20,000,000 active mobile banking users during a peak load time. We then estimated the large banks transaction rates by extrapolating from our regional bank. With this, we determined that the average TPS during peak time would be 2,915 TPS, and more importantly, the maximum transaction rate that they might see would be 11,228 TPS. Therefore, we determined that we wanted our test application to be able to scale up to ~11,228 transactions per second. That is, we wanted to demonstrate that the application provided the same response times at 11,000 TPS as it did at 3,000 TPS.
Our Test Application
Our test application was designed to simulate the types of functionality available in a mobile banking application. The application has multiple screens including a login screen, and accounts overview screen, and then account details screens. Our load testing harness included JMeter for simulating users interacting with the application. We created a user-interaction test that included 15 requests/transactions per run. Each simulated user ran this test sequence. Initial testing showed us that each run took approximately 15 seconds so each user was hitting the application with 1 TPS. Thus, in our scenario, 1 User = 1 TPS.
The back-end of our mobile banking application was designed with a simulated ESB service that would represent queries out to the bank’s back-office systems. We had this back-end service simulation include a 500ms wait-time to approximate this work.
Our mobile banking application architecture consisted of two components, a front-end component running on multiple OutSystems front-end servers with a load balancer, and a back-end component simulating the ESB. Our front-end application also accessed databases within the OutSystems platform.
Our Test Plan
Our testing plan included three different load tests at different transaction volumes.
The first load test was designed to represent a moderately busy day at our very large bank with 3,750 transactions per second flowing through the mobile banking application.
The second test simulated a surge in traffic on this busy day at the very large bank. This shoots the transaction load up to 7,500 TPS.
Finally, we wanted to simulate the most traffic that the very large bank would experience, say, for example on Black Friday in the US. Under these peak conditions, the mobile banking application is hit with 11,250 transactions per second, a very high transaction rate.
The Result – Low-Code Scales
In preliminary testing, we deduced that our test application running on a single OutSystems front-end server could handle a load of 750 users (or 750 TPS). Therefore, for our first test run, we configured five OutSystems front-end servers to handle the anticipated load of 3,750 TPS.
In our first test, we ramped up the user load linearly and then held steady at 3,750 TPS. The chart here shows the transaction response time for the 30 minute test run period after the initial ramp up.
We see that the transaction response time holds steady with an average steady-state transaction response time of 419 milliseconds, which is a very acceptable transaction response time for a mobile banking application.
With Load Test 2, we scaled up the number of front-end servers to 10 and ramped up the load to 7,500 and held it steady for 30 minutes. We see again that the average response time for this run is a very reasonable 393 milliseconds.
Finally, under the Black Friday peak load, the transaction load shoots up to 11,250 TPS and to handle that load, we increased the number of front-end servers to 15. And guess, what? For this Test Three, we achieve an even better 372 millisecond response time from the same mobile banking application running on the same OutSystems low-code platform.
So, we see that with three different levels of load on the application, we get consistent application response times.
Constant response time with increasing load is a hallmark indicator of a highly scalable application platform.
We saw through these tests that the application response time of a mobile banking application running on OutSystems would stay constant even as the load was ramped up to very high levels.
This test has shown that the OutSystems platform is able to handle very high transaction rate loads on mobile applications including the estimated load of a mobile banking application for a very large US bank on Black Friday.
So, how do we do this? How do we achieve this type of consistent performance regardless of application load?
In addition, This standard native code is executed directly in high-performance application stacks. OutSystems runs the underlying code natively. We don’t use interpreters that can slow the performance of execution.
The OutSystems platform is designed with optimized communications, high-speed load balancing, and vertical and horizontal scaling.
The good news is that you can get this same scalable platform whether you run your applications on our Cloud offering or on your on-premises infrastructure or private cloud. Your choice.