Solution, Deployment and Daily Performance Issues.

Solution, Deployment and Daily Performance Issues.

  

Hi Everyone,


Currently we have 3 years + running project with almost 1800 Applications objects.

Basically a Multitenant, In Cloud, Version 10X, we have library module (forge + javascript etc), main module that consist main db, syncronizer db, configuration and tenant setting etc, and 10 modules that reference all of those.

At first 3 years we have only few developers (mostly 6) with similar timezone.

Now in the past few weeks we have developer from another branch and with significant timezone difference.

Sydney, Perth, Malaysia and Poland to working on the same projects.


The issue now is Performance.


1. We only have 5 hours (1am Malaysian time to run solution and deployment to QA until the Sydney team online) and it took arround 4hours to finish both of it. That is not full solution, already remove some of the espace that rarely updated from the solution. But still sometimes if failed because of 'someone still makes changes' or 'some reference issue'. And when it failed I need to see if the time is enough for another solution run, else the next day the earliest timezone will be having issue until solution run finish.


2. If no Solution run and there is changes in Main module/app. The environment is really slow and we keep getting the devs and the tester complaining if they cant debug or took 5 minutes to refresh the screen something like that.


3. Issue with Espace merge and publish took long time. These are common lately because sometimes there were 2 or 3 people working on the same Espace. And yes, Manage to workload better, we already done that but its hard when the stakeholder wants changes on the similar module or the tester raise defects on the same module or espaces.


Do you guys have any advice for this?

I tried to research 

1. Warm-up to for the 1st screen performance issue after deployment, but currently I still try to figure out how to make it work as currently we not allowed to access Systems Entity from the Querries.

2. Automatic Deployment, I'm still unsure if this would be any help.


We are running really bad sprint lately because of this performance issues.

Really hope to hear some advice or solution from you guys for this.


Thank you in advance.

Hi,

Are you using life time to deploy between environment or are you uploading solution from service center? First is much better than the second.

In any case, ot seems that you have a serious problem of architecture.

In an application with 1800 AOs, having only 10 modules for the general interface seems little. How many user stories are you dealing in each module? How many screens? 

How is your segregation between end user interfaces and core business?

How many circular references? Upward references? Lateral references in the end user layer?

Cheers

Eduardo Jauch

Regarding the "slowness" in development, can be many different things.

Insufficient resources in the server (too many people connecting at the same time), bad server configuration, bad internet connection...

Hi Eduardo,


Currently those 10 module consist 7-15 espace each, 1 module is like the lowest one (Reporting Module), as it refer to all DB in main module and the rest of 9 modules.


"How is your segregation between end user interfaces and core business?" :

Library and then main module basically consist main db that consist those configuration and tenant related entity, Main user provider, main Webservices consume that will be used by the rest of the modules and common UI that handle those theme UI login etc,

And then we have 10 modules that consist DB, UI, webservices that will be consumed by main WS.


"Insufficient resources in the server (too many people connecting at the same time), bad server configuration, bad internet connection..." :

 In cloud environment how we should address this?
For Internet connection, I think I got same complain, whether it was in Office, Home, Sydney, Perth, Malaysia, Poland, We get this complain if the environment is really slow (when we failed to run solution)

"How many circular references? Upward references? Lateral references in the end user layer?"

We handle it quite well with Discovery

In an application with 1800 AOs, having only 10 modules for the general interface seems little. How many user stories are you dealing in each module? How many screens?

Previously we can handle 20 US with only 4-5 Devs (roughly 2-4 modules 8-12 espaces to change)  and currently we are struggling with 12 Devs dedicated to US (Not even 30 US) and 2 Devs dedicated to Old Defects or UI changes. We are trying to achieve some target here on the next year, but it seems bad now. We struggle with finishing sprint, which is never this worse before with lesser devs.

Hi,

When you say "module" you are talking about "Application", right?

Regarding issues with the environments, if they are Enterprise Cloud environments managed by "OutSystems", you should talk to the support team to try to figure why it is slow.

Taking 4 to 5 hours to deploy a solution means that you are probably republishing everything. This happens, most of the time, when you are changing something that everybody depends on. You must avoid this at all costs, or you will be facing this issue every time a small change in this module (espace) is done.

Are you using life time to control the deploy?

Eduardo Jauch wrote:

Hi,

When you say "module" you are talking about "Application", right?

Regarding issues with the environments, if they are Enterprise Cloud environments managed by "OutSystems", you should talk to the support team to try to figure why it is slow.

Taking 4 to 5 hours to deploy a solution means that you are probably republishing everything. This happens, most of the time, when you are changing something that everybody depends on. You must avoid this at all costs, or you will be facing this issue every time a small change in this module (espace) is done.

Are you using life time to control the deploy?

I use the Solution in service center to make sure all reference in espace is all Ok.
And then do deployment to QA env with lifetime.

Running solution is our problem.

It took us 3 hours for around 130 espaces.


Hi Eduardo,


"When you say "module" you are talking about "Application", right?"

Yeah, it just our term, hehehe sorry.

Module in our sentence is App.

Well, in a normal environment we took like maybe 10-30 seconds to publish a module.

If you have problems connecting the environment (that seems the case, for some reason), and assuming you would take between 1 and 2 minutes to publish a module, 130 modules, 3 hours seems reasonable (in the sense is something I would expect)...

I remember publishing solutions with 5 modules that took more than 5 minutes to publish, as all the modules must be refreshed, uploaded (if it is the case) and so on...

If you don't have problems with circular references and so on, why do you do this step instead of tag and deploy directly into Life Time? It will know what must be refreshed and you will avoid having to publish ALL your modules...


Cheers,
Eduardo Jauch

Make sure you use Discovery (and for example Ziggurat) to gain more insight how the references are.

The more references each one has, the more time it takes to publish and synchronize everything.

Are you actively monitoring thee references, remove unused elements is a must etc.


Another suggestion is to follow this guidelines:

1. Don't pass to quality EVERY day (twice a week, for example). This will guarantee the testers will have enough material to test to the next deploy.

2. Create solutions that have only the required modules to be refreshed.

I'm not seeing any other way of decreasing time spent with references refreshing and deployment.

Cheers,
Eduardo Jauch