Questions about 4-layer architecture

Questions about 4-layer architecture

  
Becasue we want to start using the proposed 4-layer architecture (on last nextStep), we have some questions ...

- How to work with styling, in layer 1 (infrastructure) ?
- Where to put login screen in layer 1 (infrastructure) or layer 4 (UX orchestration) ?
- Where to put layout_normal layer 1 (infrastructure)
- What will be the content of composite applications ... only webblocks or also webscreens ?

As you can see, lots of things unclear :-)
Hey guys, come on, you must have an idea how to work with this don't you !
In my experience, every time I see someone attempting this kind of architecture in Agile Platform, it is an abject disaster.

You always end up with piles of circular references, making deployments a real headache, and working is a total hassle because everything you do involved redploying to the server and then a refresh of references.

Furthermore, I see little actual benefit to such a beast. The best hoped-for benefit is to make it easier for multiple developers to work at once, but the reality is that every change requires touching most of the OMLs anyways, so you still have lots of merging going on.

In a nutshell, these kinds of architectures add significant amounts of "friction" to the projects I see them on, often reducing developer productivity my 10% - 20% in my estimation.

J.Ja
That said, if someone can show me how to make this work, and NOT have the "friction", I'd love to see it because the promised benefits are fantastic. I just have never seen them materialize.

J.Ja
J.Ja I think I agree with you on some points.
At this moment we're struggling to get it right.
We do understand the first three layers, but still don't understand how the UX orchestration layer is supposed to work.

What I can't believe is that there is nobody of Outsystems who reacts on this post :-)
They presented this on the last NextStep
Hi all,

First of all J.Ja I agree with you to some degree but let me explain why. The reason why we shared this information is based also on experience. 
We've seen many projects and many teams and we've seen them struggle and we struggled with problems related with cyclic references and re-factorization. And that also killed the productivity. Adding some structure to the architecture design can help but it also can cause "friction" just because it is a process in itself. But I really think that the benefits come higher. Nonetheless what we presented in NextStep is not prescriptive and it is not the Holy Grail of architecture design.

Benefits of having a structured approach to architecture design:
  • No wheel reinventing - once you start using this approach and you get used to it, it comes out naturally and each time you start a new project or to design an architecture you have a process to help you out.
  • Helps you think - once you go up to the board and draw three lines defining the four layers and start scattering the components it makes you think about should and lay this here or there.
  • Helps keeping the code tidy – smaller components are easier to “digest” than big ones right? Smaller components create smaller footprints speeding up deployments.
 
 Let me give you an example and walk you through the layers:
4-layer architecture
Infrastructure Services - In this layer you should have the highly reusable components. This includes: UI Patterns, connectors to external systems, external libraries and entities form other data models. @Joop this is where you should have a base theme for instance. Be careful to avoid referencing components in the layers above since this can easily lead to cyclic references.

Core Business Services - In this layer you want to isolate business services that can be shared by different composite applications. In this example you can have both OrderManagement and CRM sharing the Customers core entity. This is better than CRM referencing OrderManagement since they don't actually depend on each other and this also creates a smaller footprint for the composite applications. Normally in this layer you can have back office UIs to manage the core entities. These UIs can be web screens or can be web blocks used to mash up in layers above.

Composite Applications - This is the layer were business comes together; if you want it's the first service mash up layer. In this layer we have the business logic to support the user stories or use cases and that can’t be isolated for common reuse. It is also in this layer that we have the workflows to orchestrate the use cases. It is common that in this layer you also have entities that extend or correlate Core Entities.

UX Orchestration – In this layer we essentially concern about improving the user experience. This is where we usually have the single login page, the landing page for the role, including dashboards. At this level we can also have cross application workflows.

This last layer doesn't exist all the time and the fact that its is the UX layer it doesn't mean you should be concerned about UX in the composite application layer.

Also remember that this 4-layer was framed in a 3 step process to architect a great solution:
  1. Disclose - disclose business needs and embedded features thru user stories, information architecture, integration technology and ux expectations.
  2. Match - match the disclosed features against the known patterns and map them in the 4 layers.
  3. Architect - define component inter-communication and services.
Don't forget to read Francisco Menezes' presentation from NextStep that you can find here.

I hope this helps in the discussion.
@Joop sorry for the late answer but it took me a while to prepare it :)

Cheers,
André Vieira


@André Vieira

Youve deployed "UI_Common" under Infrastructure services, but what if your "Login" method (action) has functions that uses entity and methods in an eSpace under Core Business Services?

Such as Login method, might check if the user has created a profile under the "Account" eSpace (located under Core Business Services), if no profile has been created redirect user to create and update profile screen, if user profile has created then log the user into the application and, record the login activity. (this is just an example of cross referencing)

so now you would have cross reference - Infrastructure services reference methods in Cross Business Services (for CheckProfileActive,LogUserHistory etc), and Core Business Services references Infrastructure eSpaces (for web block styling).

One solution to this problem.... is you might create an independent eSpace to use as bridge between the eSpaces and avoid cross referencing.

In the real world its very complex! 

@Robert imho the login-action should not be in the UI_common...

@Andre any case, besides a nice discussion about the new 4-layer draft.
I would like to see an example solution/espaces of that draft :)
@Joost, ... you would think login is an infrastructure service? :/ .. in this case its a core business service, ...this should work.

-------------------------------------
CheckProfileActive (Core Business Services)
LogUserHistory action (Core Business Services)
Login action (Core Business Services)
-------------------------------------
Login screen (Infrastructure Services)
Andre -

This is a good pattern for people building out internal, enterprise applications, where there is a lot of opportunity to reuse functionality.

For the applications I work on (typically SaaS applications), this doesn't work because there is very little component reusability, so seperating things out like this costs a lot of additional time, but doesn't deliver many real benefits because so little functionality can ever be reused.

This is basically trying to take the concepts of OOP and applying them to Agile Platform, and OOP is only useful when the model gets reused, otherwise it's a lot of extra work without providing a reward.

That said... I beleive that with some changes to how Agile Platform handles certain things, this pattern could be made much, much better. The changes that need to be made:

* Allow changes in one OML to be reflected in the other OMLs in the editor without publication, and preferably automatically; make working across multiple OMLs as seamless and easy as working within the same OML.

* Make deployments from Service Studio work the same way as deployments via LifeTime, so that circular references are not a major headache like they are now.

* Have Service Studio communicate with the Platform Server to notify developers when someone else is working on an Action, Screen, Entitiy, etc. that they are working on too, so that developers do not step on each others' toes.

J.Ja
Nice to see something happening here :-)
We've implemented the first three layers without circular references.
We still don't fully understand the usage of the fourth layer. As in the picture it only shows intranet ...

@Andre : in core business services there can't be any webscreen. How to work with menu on that layer ... 
@Robert: loginscreen is in our app on layer composite applications ... it's not possible to put it in lower layers. IS layer does not contain any webscreen

I'll try to put a picture together how we did it
Ironman wrote:
@Robert: loginscreen is in our app on layer composite applications ... it's not possible to put it in lower layers. IS layer does not contain any webscreen
 
 

You are right. I should have checked that before posting! 
Ironman wrote:
Nice to see something happening here :-)
We've implemented the first three layers without circular references.
We still don't fully understand the usage of the fourth layer. As in the picture it only shows intranet ...

@Andre : in core business services there can't be any webscreen. How to work with menu on that layer ... 
@Robert: loginscreen is in our app on layer composite applications ... it's not possible to put it in lower layers. IS layer does not contain any webscreen

I'll try to put a picture together how we did it
 
 Hi Joop,

In Core Business Services you may have webscreens, typically some back office to manage the core entities. If you're using the application switcher you can even have a menu for that backoffice only and use the application switcher to change to another composite application or to the intranet at the UX layer.
For these back office webscreeens you can still have a common theme that's referenced from below in the Infrastructure Services layer.
Regarding themes also remember that you can have a base theme with the layouts defined at the Infrastructure Services layer and you can extend it at the composite application layer in order to have specific exception handling for instance.

Cheers,
André
Joost Landgraf wrote:
@Robert imho the login-action should not be in the UI_common...

@Andre any case, besides a nice discussion about the new 4-layer draft.
I would like to see an example solution/espaces of that draft :)
 
 HI Joost,

You can easily map this drawing to eSpaces/extension like so: everything apart from SAP_FI is eSpaces, SAP_FI is an extension with BAPIs to read/write information from SAP.
Robert Chanphakeo wrote:

@André Vieira

Youve deployed "UI_Common" under Infrastructure services, but what if your "Login" method (action) has functions that uses entity and methods in an eSpace under Core Business Services?

Such as Login method, might check if the user has created a profile under the "Account" eSpace (located under Core Business Services), if no profile has been created redirect user to create and update profile screen, if user profile has created then log the user into the application and, record the login activity. (this is just an example of cross referencing)

so now you would have cross reference - Infrastructure services reference methods in Cross Business Services (for CheckProfileActive,LogUserHistory etc), and Core Business Services references Infrastructure eSpaces (for web block styling).

One solution to this problem.... is you might create an independent eSpace to use as bridge between the eSpaces and avoid cross referencing.

In the real world its very complex! 

 
 Hi Robert,

I think you already got the answer to this one. You'd have your single login at the UX layer or at least at the composite application layer. The UI_Common provides common theme and common exception handling, but the redirect to the login screen should be to outside of this module.

Cheers,
André
Justin James wrote:
Andre -

This is a good pattern for people building out internal, enterprise applications, where there is a lot of opportunity to reuse functionality.

For the applications I work on (typically SaaS applications), this doesn't work because there is very little component reusability, so seperating things out like this costs a lot of additional time, but doesn't deliver many real benefits because so little functionality can ever be reused.

This is basically trying to take the concepts of OOP and applying them to Agile Platform, and OOP is only useful when the model gets reused, otherwise it's a lot of extra work without providing a reward.

That said... I beleive that with some changes to how Agile Platform handles certain things, this pattern could be made much, much better. The changes that need to be made:

* Allow changes in one OML to be reflected in the other OMLs in the editor without publication, and preferably automatically; make working across multiple OMLs as seamless and easy as working within the same OML.

* Make deployments from Service Studio work the same way as deployments via LifeTime, so that circular references are not a major headache like they are now.

* Have Service Studio communicate with the Platform Server to notify developers when someone else is working on an Action, Screen, Entitiy, etc. that they are working on too, so that developers do not step on each others' toes.

J.Ja
 
Hi Justin,

I think reusability is only one of the plus about this approach and I agree with you that this fits better with enterprise grade applications. Nonetheless even if you're doing SaaS you still need to identify the modules composing you're application.

Regarding the overhead you mention.... even if there's some overhead (which I think isn't that important) the platform already helps you out. The first change you'd like to see implemented in the Agile Platform already exists, you can refresh references in open eSpaces without publishing, offcourse when you want to test it you'll need to publish it. You can even create new references using drag and drop.

In this first picture you'll see that it is possible to refresh a changed reference without publishing.


In this second picture you'll see that it is possible to define a new reference through drag and drop.


I dragged and droped the PublicAction action from Customers eSpace to Cases eSpace.

The circular references is a problem like the chicken and the egg, it is better to avoid it :)

Cheers,
André




André Vieira wrote:
In this first picture you'll see that it is possible to refresh a changed reference without publishing.


 

Unfortunately this option is not always available.

Example you added a web screen to your producer eSpace, it is not related or affects any of the consumer eSpaces, however you can not simply click on refresh the consumers eSpaces or am I doing something wrong here?
Robert Chanphakeo wrote:

Unfortunately this option is not always available.

Example you added a web screen to your producer eSpace, it is not related or affects any of the consumer eSpaces, however you can not simply click on refresh the consumers eSpaces or am I doing something wrong here?
 
Hi Robert,

I'm not totally following you....
In order to refresh you need to have references to it already. In that case you say it is a new web screen so what I think you want is to add a reference and not refresh it so you'd use the second approach to drag and drop from one eSpace to the other. Remember that the object must be public in order to be able to complete the droping of it in the destination eSpace.

Cheers,
André
André Vieira wrote:
 
Hi Robert,

I'm not totally following you....
In order to refresh you need to have references to it already. In that case you say it is a new web screen so what I think you want is to add a reference and not refresh it so you'd use the second approach to drag and drop from one eSpace to the other. Remember that the object must be public in order to be able to complete the droping of it in the destination eSpace.

Cheers,
André
 
 @Andre

What I mean is if your consumer eSpace reference say an "action" from your producer eSpace, now you added a web screen to your producer eSpace which is not at all affecting the consumer eSpace you still have to refresh the consumer eSpace, because the producer eSpace changed, I know why this happens technically and why its required to refresh and republish.

EDIT: refresh does not actually refresh and deploy! it just refresh all consumers that are open consumers aswell.
Robert Chanphakeo wrote:
 
 @Andre

What I mean is if your consumer eSpace reference say an "action" from your producer eSpace, now you added a web screen to your producer eSpace which is not at all affecting the consumer eSpace you still have to refresh the consumer eSpace, because the producer eSpace changed, I know why this happens technically and why its required to refresh and republish.

EDIT: refresh does not actually refresh and deploy! it just refresh all consumers that are open consumers aswell.
 
 If you add a new webscreen to the producer you don't need to refresh references in the consumer but you may still get a warning that the consumer is outdated. You just need to republish the consumer, no need to refresh references.

André
another question about it. In the presentation it was said the entities should be made read-only and you should make actions for it, so you can have control over the data. I like that very much :)
However, when you have an external-database, you cannot make those read-only.
What is your work-around for that?
André Vieira wrote:
 
Hi Justin,

I think reusability is only one of the plus about this approach and I agree with you that this fits better with enterprise grade applications. Nonetheless even if you're doing SaaS you still need to identify the modules composing you're application.

Regarding the overhead you mention.... even if there's some overhead (which I think isn't that important) the platform already helps you out. The first change you'd like to see implemented in the Agile Platform already exists, you can refresh references in open eSpaces without publishing, offcourse when you want to test it you'll need to publish it. You can even create new references using drag and drop.

In this first picture you'll see that it is possible to refresh a changed reference without publishing.


In this second picture you'll see that it is possible to define a new reference through drag and drop.


I dragged and droped the PublicAction action from Customers eSpace to Cases eSpace.

The circular references is a problem like the chicken and the egg, it is better to avoid it :)

Cheers,
André



 
 
 Andre -

Yes, I do use the module approach... it is still VERY painful.

If you think that the "cost" of using this is low, I invite you to spend an hour to watch me work on screenshare. During initial "greenfield" development, it is true that the "cost" is low. Once I get to the change/test/change/test cycle, the cost is incredibly high. I spend 80% (or more!) of the time I spend debugging... waiting for publish/deploy to occur. This is an incredible waste.

With a standard .NET application, changes get made to lots of small files which gets recompiled as needed, and it is DYNAMICALLY linked. With Agile Platform, changes get made to a few giant files, which are then STATICALLY linked. Every 10 second change requires 3 minutes of upload, compile, SQL checking, deployments, etc.

Again, I'd welcome an opportunity to have you "sit over my shoulder" to see the problems. Maybe you can show me a better way of doing things? But right now, the troubleshooting/fixing process is painful because of the "friction" caused by multiple eSpaces.

J.Ja
@André

Do you create a solution so that you can click on publish and everything would republish itself all on its own? or do you open up individual eSpaces and republish them one by one? 
Robert Chanphakeo wrote:
@André

Do you create a solution so that you can click on publish and everything would republish itself all on its own? or do you open up individual eSpaces and republish them one by one? 
 
 @Robert,

In this scenario of reference add/refresh I need to publish first the producer and afterwards the consumer otherwise publishing the consumer will result in broken references.

@Andrew

Yes, I'm aware of that, however I was wondering if you publish your eSpaces manually or automatically via ServiceCenter as a Solution.

I am asking because in the real world, you would have many eSpaces to publish, in your example if you made changes to the UI_Common eSpace, you would most likely have to republish all the consumer eSpaces above and there could be many eSpace, hence resulting in an extremely time consuming process, and lost of time.


Lets say you made changes to "Customers" eSpaces, you would have to republish "Orders", "OrderManagement" and possibility "Intranet" now your time spent republishing your application went up by 3-4 times, its going to take 3-4 longer!

In the real world you would have many more consumers and with each and every consumer eSpace it will take longer and longer to republishing your application?
 You wouldnt publish once per day, you would do this many times per day! So im wondering what you do....
 

Robert Chanphakeo wrote:

@Andrew

Yes, I'm aware of that, however I was wondering if you publish your eSpaces manually or automatically via ServiceCenter as a Solution.

I am asking because in the real world, you would have many eSpaces to publish, in your example if you made changes to the UI_Common eSpace, you would most likely have to republish all the consumer eSpaces above and there could be many eSpace, hence resulting in an extremely time consuming process, and lost of time.


Lets say you made changes to "Customers" eSpaces, you would have to republish "Orders", "OrderManagement" and possibility "Intranet" now your time spent republishing your application went up by 3-4 times, its going to take 3-4 longer!

In the real world you would have many more consumers and with each and every consumer eSpace it will take longer and longer to republishing your application?
 You wouldnt publish once per day, you would do this many times per day! So im wondering what you do....
 

 
 This is EXACTLY my point as well. Like I said, other than initial development of features from scratch (where I can go hours without a publish/deploy in many cases), I spent much of my day waiting for the work I did in one eSpace to deploy so I can use it in another. The "Refresh Consumers" does not work very well. Why? Because the application is in modules, and those modules can't be re-tested with out a re-publish of every consumer!

It absolutely needs to be changed so that consumers can target a published producer WITHOUT a full re-publish so long as the "signature" (or "prototype" for the C and Pascal folks out there) of the item do not change. For example, if I update the logic of a Web Block, but its parameters do not change, I *need* to be able to have my consumers immediately make use of it without a full publish. I am looing a ton of time every day because of this!

J.Ja
Justin James wrote:
 
.....Because the application is in modules, and those modules can't be re-tested with out a re-publish of every consumer!

It absolutely needs to be changed so that consumers can target a published producer WITHOUT a full re-publish so long as the "signature" (or "prototype" for the C and Pascal folks out there) of the item do not change. For example, if I update the logic of a Web Block, but its parameters do not change, I *need* to be able to have my consumers immediately make use of it without a full publish. I am looing a ton of time every day because of this!

J.Ja
 
 
eSpace "B" consumes, "entities" from provider eSpace "A"

Now in eSpace "A" I add a new action, web screen, I publish eSpace "A" the agile platform will now tell me, eSpace "B" is out of date.
It shouldnt because i didn't change any entities, that would affect eSpace "B", and its not referencing any actions or web screens in eSpace "A" is it? so it shouldnt tell me to republish.

The agile platform knows eSpace "B" is not affected, but technically it requires republishing because the producer eSpace "A"'s dll has now changed.

It can not republish on its own or does it really need a developer to look at unaffected eSpaces before republishing?
Justin James wrote:
 
It absolutely needs to be changed so that consumers can target a published producer WITHOUT a full re-publish so long as the "signature" (or "prototype" for the C and Pascal folks out there) of the item do not change. For example, if I update the logic of a Web Block, but its parameters do not change, I *need* to be able to have my consumers immediately make use of it without a full publish. I am looing a ton of time every day because of this!

J.Ja
 
Definitely got my vote. No signature change should leave consumers marked as valid, otherwise there's a whole load of manual work trying to figure out the right order to do loads of manual re-publishes to keep your dev server looking correct. Without it, we lose what's normally the benefit of OutSystems and its simple deployment process.
Iain Fogg wrote:
 
Definitely got my vote. No signature change should leave consumers marked as valid, otherwise there's a whole load of manual work trying to figure out the right order to do loads of manual re-publishes to keep your dev server looking correct. Without it, we lose what's normally the benefit of OutSystems and its simple deployment process.
 
 Unfortunately when you publish an eSpace and make changes to the web screen (or any other item), the dll is changes! So if your consumer reference any other items such as maybe an action, you technically would still need to republish the consumer eSpaces. - Maybe in the future, outsystems could automate republishing of consumer eSpace?

For now, will just got to make do!  :-)

Sure, but if I install an update e.g. a hotfix to .NET on my PC, I don't have to reinstall or update every .NET application. They can carry on using the existing DLLs where the interface to that DLL is the same. We will have to make do, but going forward I think this is key to having a more effective and efficient way to do multi-level applications.
Iain Fogg wrote:
Sure, but if I install an update e.g. a hotfix to .NET on my PC, I don't have to reinstall or update every .NET application. They can carry on using the existing DLLs where the interface to that DLL is the same. We will have to make do, but going forward I think this is key to having a more effective and efficient way to do multi-level applications.
 
 Yes you are right! :)

.NET you dont have to recompile everything, in agile platform you do, at least for now until its fixed?