Building a Fort: Security Through Automation in OutSystems Sentry

In the Building a Fort OutSystems Sentry Series, we’ve shared our compliance process in The OutSystems Sentry Security Compliance Process, our technical challenges in The OutSystems Sentry Toughest Technical Challenges, and our ongoing monitoring in The OutSystems Sentry Monitoring. The topic of this blog post is how we came to the concept of security through automation in OutSystems Sentry and how adopting it helps us maintain a healthy and secure PaaS offer.

Securing the Cloud

Delivering OutSystems Sentry meant implementing several non-functional requirements in our existing PaaS, such as compliant audited operations and infrastructure security controls. Identifying these requirements created a clear picture that would not only ensure high levels of compliance and meet OutSystems excellency standards, but it would also make things easier for our operations team. Developing automation for a secure product required a design process that was more focused on security. To this end, our team set up mandatory threat modeling when designing new solutions to identify new assets, entry points, threats, and finally, mitigations strategies.

The main targets for automation were server configuration hardening, network security, and temporary operational users. For the latter, check out Miguel João’s Building a Fort: The OutSystems Sentry Toughest Technical Challenges for more details.

Server Configuration Hardening

An initial analysis of the existing machine configurations and documentation requirements indicated that for an existing customer to become a Sentry customer, we would need to have a perfect history of all operations on the machine and a foolproof hardening that could take into account every possible configuration. After evaluating this approach with threat modeling in mind, it proved to be too costly to implement, both in effort and risk.

Ensuring our Sentry customers’ utmost security meant making a bold decision: there had to be a clean slate to ensure total control from that point on and that every change is audited. Thus, we would need to re-create all the customers’ servers to a new template that had all the necessary configurations.

Afterward, we needed to make these new templates as secure as possible. After a review of the industry standards for server hardening and best practices, we chose CIS Benchmarks for the operating-system and web-server configuration because they offer detailed recommendations. Because OutSystems is a low-code platform that provides the valuable configurations our clients want, we must always analyze security recommendations to find the most secure approach without compromising the platform's flexibility. Working with these constraints allowed us to produce an extensive document that details what is configured, what isn’t configured, and why, which is a cornerstone for compliance. Having this information also meant that using threat modeling became easier and allowed us to iterate and improve until we reached the OutSystems excellency standard we strive for.

Although this type of “big-bang” solution tends to be avoided, in hindsight, it was the perfect approach. We were able to create extensive quality documentation when hardening the servers from scratch, which was valuable for proving compliance and maintaining the templates. Not only that, but other areas in the company were able to harden clients’ servers based on this documentation.

Tightening up the Network

To get network security right, every type of communication is of the utmost importance. After gathering the requirements of the services used in our infrastructure, we created an inventory of the necessary accesses a certain type of machine requires, including the port, source, and protocol (which can be dynamically or statically defined).

We hadn’t finished the requirements analysis at this point because our customers had requested customizations to this network configuration over time that we needed to maintain. Knowing this, we designed an architecture that uses automation for maintaining both sides of the configurations and is properly documented. Additionally, to ensure security sanity, we became an authoritative entity, knowingly discarding any other unknown configurations in the managed system. This helps protect against rogue changes to the configuration and enforces auditing by design.

We used two layers to implement this architecture: AWS Network Security Groups and Firewall software in the servers. This was a decision driven by our threat modeling enforcement for all aspects of this project; ensuring a two-layered approach protects our customers from intrusion unless both AWS and Server assets are compromised. 
When someone adds or changes a rule in our back-office, both layers are automatically synchronized with proper security checks and auditing.

To support our customers’ needs, we came up with two types of network rules: system or customer. System rules are defined by the technological and infrastructural requirements of the PaaS during feature development. Customer rules are customizations they request to enable interactions between their applications and users that meet business needs. Using this segregation between customer and system rules, we develop new features faster as customer network rules are separated from the infrastructure.

Getting this architecture up and running was a challenge. It took considerable research to learn all the necessary rules and a conversion step between the old and new.
Most importantly, however, our clients should not be affected at all, as it could have catastrophic consequences. Fully deploying this architecture to about 2500 machines and 950 databases took about 20 hours between two shifts of the team due to how critical it was. In the end, the conversion was a success. From that point on, the network configuration became fully managed automatically with an audited user interface for operations to configure new client-requested rules.

Keeping the Guard Up

With threat modeling as part of our development process, we can deliver features with security through automation. This ultimately allows us to provide valuable features to our customers which are secure by design.

Being able to participate in these complex and challenging projects and deliver unquestionable value to our customers is what makes me motivated to work at OutSystems.