This post provides a compilation of best practices for non-production systems used by big factories with large development teams (over 30+ developers). This information is also applicable if you have been using your OutSystems platform for some time and notice performance degradation.  

You can apply these best practices when developing your OutSystems applications and maintaining your non-production environments. These best practices are recommendations meant for you to adapt to your environment and your development style. 

The following topics explore different ways to fix a problem:

• Keeping your environment clean - a clean environment helps make your system more efficient.

• Tuning - for the best system performance, keep your apps and OutSystems platform tuned.

• Defining your Team Structure and Architecture - every team works differently and it's important to define an architecture that meets the needs of your team.

• Using Personal Area - using a personal area can save you some steps during the build cycle.

• Monitoring - checking for anomalies and resolving them quickly helps to keep your apps and OutSystems platform running smoothly. 

Keeping your Environment Clean

This section applies to PaaS and On-Premises infrastructures

As a factory grows, it consumes more resources like disk space, CPU, and memory from frontend and database servers. It’s important to keep your environments tidy and clean so you don’t affect DevOps performance and don’t have to worry about incurring costs to improve the processing power and infrastructure capacity.

With a clean environment, you can eliminate a: 

• Reduction in deployment and publishing performance

• Slow down in your teams’ performance

• Negative impact on LifeTime and Service Center performance

As Factories grow and the number of modules and versions increase, consoles (such as LifeTime and Service Center) can have performance issues. These issues create a poor user experience. Since LifeTime connects to all environments, this performance issue can affect all of your environments. To fix the problem, you can use Service Center, and follow the guidance in these two articles: DBCleaner API and LifeTime API v2.

For large factories:

• Keep the number of module versions low, possibly limiting the versions to two weeks;

• Delete modules that are no longer needed, like test modules, proof of concept (PoC), or discontinued applications. 

Manage your database data:

• Reduce the amount and length of time that you keep database data;

• Delete test/dummy data from time to time;

• Create a timer that runs periodically to remove test/dummy data;

• Review query performance best practices;

• Use SQL queries for massive updates or deletes instead of using Entity Actions. Write queries that update as many rows as possible in a single statement rather than using multiple queries to update the same rows.

Use OutSystems Platform APIs

The OutSystems Platform provides some APIs that you can use to help you keep environments clear of unnecessary data:

• DBCleaner API - use it to permanently delete entities and attributes that were removed from modules. OutSystems uses a soft delete approach, that eases the rollback process if necessary, therefore, tables and columns are never removed from the database during deployment.

• BPT API - use it to remove all the logged information of an instance of a top-level Process, that must be either terminated or closed.

• Email API - use it to delete email information. This is especially important since emails can often have large attachments.

Some components on the Forge already have some of these housekeeping tasks implemented, while others also have some automated tasks.

It’s also important to manage your LifeTime users. When a user no longer needs access or leaves the company, permissions should be revised or revoked and the account should be deactivated.

Tuning

This section applies to On-Premises infrastructures because it requires access to servers and/or database. Rest assured that in PaaS this is already taken care of by OutSystems.

Every OutSystems platform server installation checklist includes tuning steps. Tuning ensures you are receiving the best performance from the server for your platform and application. The original configuration is done during installation. It is a best practice to review your configuration periodically as your applications grow. 

To tune your configuration, follow the instructions in the “Tuning your systems” section in the Installation Checklist. The checklist includes system, Internet Information Services (IIS), .NET framework runtime, and Windows Defender information.

The Performance Best Practices - Infrastructures article describes web server memory settings, virtual memory, and system settings, as well as database file growth and maintenance plans. Reorganizing or rebuilding existing indexes and updating the database statistics provides better performance in querying the database.

For IIS tuning you can find information in the Tuning IIS for OutSystems and Understanding and fine tuning Application Pool Recycling for use with OutSystems articles.

Timeout is a relatively common problem. Managing timeouts enables you to control how your applications handle longer requests. For most use cases, the default timeout values are sufficient. Review the Timeouts Under the Hood article to set up different timeout configurations. In this article, review the “SQL query timeout” and the “Application request timeout” sections. These timeout configurations may affect how your applications and OutSystems platform function. 

Defining your Team Structure and Architecture 

This section applies to PaaS and On-Premises infrastructures

When you add more developers, teams, and applications, it’s important that you remain agile. To continue to develop apps in an agile environment depends on your team's ability to be independent but also align with business goals. 

When teams are independent, their delivery speed increases. Being independent is beneficial when it comes to development delivery alignment with business goals, and maintaining a loosely coupled architecture. 

The development and delivery at scale article focuses on some aspects (domain-driven design, team structure, ownership and governance) of the factory architecture and organization that must be in place for effective team isolation.

The way applications are designed and architectured significantly impacts scalability. The Architecture Canvas is an OutSystems tool to make the design of Service-Oriented Architectures (SOA) simple. When developing and maintaining multiple applications that reuse common modules, SOA promotes the correct abstraction of reusable (micro)services and the correct isolation of distinct functional modules.

A Domain Driven Design (DDD) is a way of developing complex software by translating it into small connected pieces in an ever-evolving model of the core business logic. It is a methodology and process prescription for the development of complex systems. The focus of DDD is mapping business processes within a problem domain into the technology artifacts of a solution domain. To learn more about how to create a domain architecture based on a Domain Driven Design using OutSystems, click here. 

Another aspect to consider is module references. As explained in the Performance Best Practices - References article, using public actions improves application performance. But, public actions increase the complexity of the application’s reference map. The complexity of the reference map then hinders the performance of the deployment process. 

Before deciding between web services and public actions, and the benefits of good application architecture, think about segregating work teams/developers. Should you put teams/developers on the same or different modules? If you have several teams working on the same modules, this may cause workers to:

• Clash with other workers;

• Merge their code constantly;

• Wait for others to deploy their code before a worker can deploy their code;

• Update the entire reference hierarchy.

The Architecture Dashboard is an OutSystems tool that provides you with an overview of your organization’s technical debt across its application portfolios. The Dashboard shows the details of the technical debt of the apps in the infrastructure. You can see the apps that need fixing. Then you can take the steps to reduce the debt score for the problem areas/applications/modules.

Focusing on the infrastructure level, there are other decisions that you need to make. Consider balancing the load between servers by: 

• Having a dedicated database (DB) server for your development environment

• Adding more frontend (FE) servers to your development environment

• Having a dedicated Deployment Controller (DC) server

In a development environment, the DC server is used every time a developer publishes a module, application, or solution. For more information about infrastructure sizing, review the Sizing OutSystems Platform article. However, keep in mind that currently none of these infrastructure changes are available on OutSystems cloud offer.

Another option is to work with pipelines since each pipeline has its own infrastructure. Pipelines enable you to develop independently and release one or more related applications. Pipelines can help you with governance, compliance, and the maintenance of standards by enabling you to:

• Isolate lines of apps with different lifecycle stages into pipelines. Each pipeline can have a different set of environments.

• Group applications with similar scalability and high availability needs.

• Address compliance with data segregation policies with separate databases for more sensitive data.

• Isolate mission-critical apps. Pipelines weren't designed to isolate each and every application. But, mission-critical core systems that support a substantial part of your business may require their own pipeline. You might use a pipeline not to just ensure the app is not impacted by noisy neighbors, but also because your app may require additional lifecycle stages and data isolation.

For example, you can have one pipeline for the apps of each line of business, or one pipeline for a critical business application. The downside of pipelines is that they require different infrastructures, which increases costs. As multiple infrastructures work independently from each other, management tasks like LifeTime users and permissions create more effort for you.

Another disadvantage of using pipelines is that governance over common/generic modules that you might want to use across multiple pipelines can be more difficult. Nevertheless, you can reuse modules by exposing APIs for the required features. If you reuse modules, make sure you implement proper security measures.

Using Personal Areas

This section applies to PaaS and On-Premises infrastructures, but only to traditional web applications.

A personal area is a place where a developer can test changes to web application modules without affecting the work of other team members in the same modules. This lets you publish your code without publishing it in the public area.

When you deploy a module using 1CP (1-Click Publish), a new version of the module is stored in the environment database, keeping it under version control, and the code is deployed to the public area of the environment. All changes become public to other developers and users. If other developers are working on the same module and publish their changes, their work merges with your changes.

Instead of using 1CP to deploy changes, you can use the option titled “Run and Debug in [user name] Personal Area”.

When developers publish a module to their Personal Area their work is private. This area is not visible to other developers or consumer modules. 

When publishing to a Personal Area a new version of the module isn't created in the database. Since there is no version control, the database uses fewer resources. Also, there are some changes you can't make. For example, you can’t change entities or site properties. Since you can’t make some changes when using your personal area, some publishing process steps are skipped. So, the Deployment Controller Services use less processing time and resources.

Be aware that:

• Personal Areas are only available to traditional web applications

• Shared services like Processes, Timers, and Emails only execute in the Public Area

• Some changes require you to publish the module to the Public Area to ensure consistency between the Public and the Personal Areas.

To learn more, see the Public and Personal Areas article.

Monitoring

Infrastructure monitoring applies to On-Premises infrastructures because it requires a local user account to get access to metrics. Those metrics are monitored by OutSystems in PaaS infrastructures.

Application monitoring applies to PaaS and On-Premises infrastructures.

Monitoring enables you to check an environment's availability, spot bottlenecks, and measure overall performance. Monitoring should be done at two levels: infrastructure and application.

Infrastructure-level monitoring considers the ecosystem’s hardware, operating systems, application servers (IIS), and database servers. The following parameters should be monitored:

• CPU 

  • CPU usage 

  • Processor queue length

• Memory

  • Available memory

  • Pages read/write ratio

• Disk drives

  • Disk queue length

• Network

  • Bandwidth usage 

  • Errors

• IIS

  • Requests queued

• Database Server

  • Monitor CPU, memory, disk drives, and network as above mentioned

  • Buffer cache hit ratio

  • Page life expectancy (applicable only to MS SQL Server)

  • Batch requests

  • SQL compilations

  • SQL Re-compilations

  • Lock waits

  • Read and Write I/O operations

For on-premises infrastructures, you can use the Infrastructure Monitor, a LifeTime component that provides some of the metrics mentioned above, among others.

Application-level monitoring focuses on application errors and performance. OutSystems provides built-in monitoring and analytics tools that enable proactive management of application performance to make it easier to detect, identify, and troubleshoot issues in real-time.

These tools are automatically available through the platform, and all OutSystems applications collect logs and performance events and store them in a standard relational database. These metrics are collected by default, requiring no extra work on the part of developers and adding no overhead to the runtime environment.

Service Center provides monitoring logs and analytics reports about performance issues of the platform and applications. It is important to take into consideration also errors and warnings (such as SLOWQUERY) from the platform itself and its system components. These logs are associated with the (System) module.

LifeTime provides an aggregated view into Web application performance and end-user experience.