Aggregates and Simple Queries

Aggregates and Simple Queries


Is there any good reason why platform 9 can not have both Aggregates and Simple Queries alongside ?

This way, situations where simple queries are ok, stay with simple queries (including migrations!!!) and other situations where Aggregates are better, go to Aggregates.
Would this be an option in a near, near future ?

Hi Pedro,

That's not on our plan, but we'd sure like to know in which situations do you consider the pre-9 queries to be a better fit.

In my opinion the aggregates is much better solution, however would be very nice if we could get the "backstand" SQL, at least to see it.
Hi Alberto,

The facility to view the SQL code that was executed wasn't production-ready in time for the Platform 9 release but it should be back in a (very) near future.

Hi Miguel,

As for your question why one would consider pre-9 Simple Queries better, I have to say that I just totally hate the way aggregates are displayed. A Simple Query, especially the truly simple ones, like getting the records of a static entity, or get a few records with a table or three that are linked straight forwardly, can be seen in one glance. You can just look at it and understand what it does. With aggregates, I have to click through three tabs to see the same information! Also, since for some reason the interface is changed to full width instead of inside a window, I can't have two aggregates side-by-side to compare them easily (if "easily" is the right name to start with, given the above), and on my 27" monitor it's a huge waste of screen real estate. I'm really wondering whether this isn't a case of usability being totally forgotten in favour of fancy technology. I really hope some bright mind at OutSystems can redesign this complete mess, and make it into something usable (from a user's perspective, no doubt it's fine technically).
Hi Kilian,

Thank you for your reply! The Aggregates are still being improved in many ways (including their design and UX) and you'll likely see some minor changes throughout 9 and some bigger changes for the next major version. We really appreciate concrete feedback from the community, like yours, that we can use to prioritise our next developments.

Hi Miguel,

Thanks for the response. I'm looking forward to future improvements!

Best regards,

Hi there,

I totally agree with Kilian Hekhuis, the way that aggregates are displayed is just a mess and very shocking for those who are used to simple queries.
Dont get me wrong, i think the aggregates have much potential and are very useful, but for simple scenarious, it would be more productive and user friendly to continue to use the pre-9 simple queries.

I really hope that Outsystems consider this in the future.

Thanks, Nuno Guedes

even though I am used to aggregates by now (even when my laptop completely locks up for some seconds when opening an aggregate for the first time) it's horrible for fast usages/testing

Imho, the main missing feature (which most developers abused the simple query for is the test-query)
The number of times i drop a simple query on the screen just to see if the data is correct, wether a join works correctly etc. is numerous.

Now I have to use an aggregate or an advanced query, both are cumbersome for that specific usage.

That's exactly what I was refering when started this discussion, Nuno!
Why not have the option to use pre-9 simple queries or Aggregates? And let it be a programmer decision, according to each different situation?
I was hoping this could be reconsidered by Outsystems in a future 9 version...

I'm glad from what I read here, because it is something that me and my team also feels, and it is good to know that others quality developers think the same as us, so I can only wish the prompt return of the excellent simple queries.

Walter Ruço
My opinion is to have both simple querys and Aggregates. I think Aggregates are very powerful but simple queries are more readable when analysing code, so when migrating from older systems and in developments one should have the option to maintain simple queries or use Aggregates. However if Aggregates where to be improved to me more readable, like simple queries, i would say definitely Aggregates.

Best Regards,
Vasco Batista.

I full agree with you Vasco!!!
Also because if I full understand from the varios inputs, the change is not only a question of UI, but also releated with entities performance. Someone correct (and inlight) me if I am wrong!!!
I agree with the Simple and Aggregates.  I do find the new UI not very intuitive or visual. It is quite confusing when adding joins.
Recently I've been hacking away at a small project in my personal environment using version 9 (at work we still use 8, though we're considering upgrading at short term), and I've been forced to use aggregates as a result. Unfortunately, my objections against them have only grown, and I can't get accustomed to them at all. I've been meaning to write down my findings earlier, but since it's quite a lot, I had to postpone. Anyway, on with the show.
First, we have three categories of issues. The first is the fact that aggregates completely replace simple queries, instead of living beside them. The second is that aggregates are a pain to use, both as replacement for simple queries and for the more serious stuff that's meant to replace adevanced queries. This is mainly a matter of a cumbersome and botched UI. Thirdly, we have some newly introduced paradigms that I don't like. I'll deal with these categories seperately.
As for aggregates replacing simple queries, judging this thread I'm not the only one who simply thinks this was a bad decision. Simple queries are just that: simple. Used for simple enumeration of entity values (e.g. for use in list boxes), simple joins (main entity plus one or two others, especially static entities), you could open them and see in one glance what's going on. Aggregates are powerful, but I don't need power for that. It's like trying to drill in a nail with a sledge hammer. Aggregates are just not the tool for that job.
Now on to the UI. Much has been said about it, but I've encountered some things not mentioned before. A list of my grievances so far:
  • First and foremost, the splitting of the query's functionality over three tabs, and having these tabs full-screen only, makes it very difficult to see what's going on. This makes aggregates very much write-only, which is bad for maintenance. Especially because the Filters tab will be populated sparingly, and the Sorting tab even more so (how many fields do you want to sort on?).
  • The "Sources" tab has way too little room to show what's going on. When having entity names in all caps and longer than about 15-20 characters, all tables look the same. Of course, one could ask why the hell one would want such long names, but our database has hundreds of tables, and too keep names meaningful without creating many duplicates we prefix them a lot. I think that's generally good practice, so I've done that in my personal project, which is a kind of dictionary app, as well. So I have LEXICAL_ENTRY, LEXICAL_ENTRY_DATA, LEXICAL_ENTRY_DATA_TYPE and even LEXICAL_ENTRY_DATA_TYPE_ENUMERATION. Now see what it looks like of I join LEXICAL_ENTRY_DATA and LEXICAL_ENTRY_DATA_TYPE for use in a combo box: A picture speaks a thousand words... Note that there's also no mouse-over showing the full name, so the only way to see it is to scroll through it, or copy/paste it. Also note that the actual join condition is hidden from view. The interface only shows the join type, but not the join condition. You have to click the Edit icon to see it (which, funnily enough, pops up an old fashioned expression editor).
  • The above also goes, and even worse, for the entity selection of the joins. Since there's no aliasing here, you can only select the entity via a list, and there's no input for scrolling through the name. See the below screen grab: which entity was selected here? The interface doesn't show the currently selected one, so I haven't got a clue.
  • Also in the Soures tab, it's possible to use aliasses for the entity names, but that's a feature with it's own issues. Suffice to say it's really weird that nowhere, afaik, it's possible to see the original entity name once it's changed. I've encountered at least one bug with aliasses, so I don't use them, but I find this a weird omission.
  • Another usability issue on the Source tab is adding a Source. With simple queries, one could drag an entity, or select an entity, and it would be added plus some automatic join conditions. With aggregates, when one adds a source via the Add Source button, the entity is added but there's no join condition created, so you have to set that up manually, which is cumbersome to do, and even more cumbersome given the interface (clicking Add Join, clicking two or three times to select the tables and the join type, clicking the add join condition button, add the join condition - yawn). Fortunately, it is possible to drag the entity from the Data tab on the left side onto the Sources tab, to get Service Studio to add the joins automatically. But there's a drawbackof course: what happens is that not only the entity itself is dragged, but also all tables it links to via entity identifiers! If that's not whay you want, you have to manually remove them again. And though I can see the rational behind it, if you have already (manually) added an entity, dragging that entity onto the Sources tab does not add a second instance of that entity, but only the tables with foreign key relations (if there are any, otherwise you can't drag at all). Sometimes I want the same table linked twice, but that doesn't seem possible (at leat with drag-and-drop).
  • On every tab, below the data specific for that tab, there's the result grid (actually, it's below the tab data - when the aggregate is opened, there's no tab selected, but the result grid is shown). This is in and of itself a nice feature, but the way it's set up is somewhat clumsy. First, when opening an aggregate for the first time, it tries to fetch this data from the database. This may take some time. If the data set is large, this takes ages. Well, at least seconds, which is way too long. Secondly, with its insistancy on keeping Sources, Filters and Sorting separate on different tabs, I find it strange, paradigmly speaking, that Service Studio allows clicking the "New attribute" button on every tab. This is a minor quible though, so let's move on.
Now, on to the third and last category, which is basically "everything else", but mostly comes down to paradigm. You see, the main difference between an aggregate and a simple query or advanced query, is that the latter two are like actions, in that they have (or can have) parameters local to the query. Aggregates don't have that: instead, an aggregate is more like the rest of the code it appears in, in that it sees all variables that are in scope. This, at first value, sounds great: all too often you're creating a list of query parameters that are mapped one-to-one to local variables or a screen's or action's input parameters, which isn't needed at all with aggragates: you can just use the local variables or input parameters directly. World domination! Or is it? This indeed sounds great, until you realize what you need to do when you don't have a one-to-one mapping, but you input, say, the output of a function directly, or the result of a calculation, or, in the case of a Refresh Query, have different input values in the Preperation and the screen action you use the Refresh Query in. In all of these cases, you need to create a local variable that's likely to be used only by the aggregate (and Refresh Query). I'll repeat that to let it sink in: in order to use something that's not already a variable in scope, you need to create a dedicated local variable that you assign whatever you want, then reference in the aggregate. In other words, instead of having local query parameters, you expose those parameters to the outside world. We have screens with dozens of simple queries, with parameters that needn't be one-to-one mappings. Some of these screens gain (through automatic conversion in the 8->9 upgrade) ten or more local variables. As a last mention on this topic, since we don't have parameters at all, the (imho) very useful "Search parameter" has also vanished. I think this has been mentioned elsewhere, but that means you need to have filter conditions like "variable = 0 or Entity.Attribute = variable". Not especially pretty, but needed in advanced queries as well, so a minor thing compared to the above.
So, that was the rant. Since I'm really basically a positive guy (really! trust me! join me at the next NextStep and find out! :)), here are some recommendations for OutSystems to fix the above mess:
  • Reintroduce simple queries. I know, that's probably way easier to say than to do, but really: they served a purpose, everyone liked them, there was really no need to do away with them. Unfortunately, this will probably come too late for many of us. Unless they are reintroduced in the next minor version upgrade of platform 9, soon we (at our company) will be confronted with hundreds, if not thousands, of converted simple queries in the 8->9 upgrade process, that can never be unconverted. I shiver at the prospect. (It would be really nice to have a "Convert aggregate to simple query" option in the contect menu of aggregates to facilitate converting back!)
  • Redesign the aggregate UI. At the least, put it in its own window, like simple query's were and advanced query's still are, so I can do side-by-side comparisons of queries. Far better, also completely redesign by putting everything (or almost everything) on a single tab, in such a way it's possible to see in one glance what's going on (at least for simple stuff).
  • Background-load the result data when opening the aggregate, to increase performance.
  • Add automatically adding join conditions when selecting, rather than dragging, an entity.
  • Add optional parameters. It's nice having access to all variables in scope within the aggregate, but there are use cases (which I've explained above) that really need them, in order to avoid having to create local variables just for that purpose.
Well that was it. My longest post ever (and I don't expect ever to write a longer one), which I hope serves some purpose other than just me getting this rant of my chest. I'd welcome input on this, both from users and OutSystems. I love the platform, but aggregates... Don't get me started. Oh wait...
I am new Outsystem user (I already start with Version 9), and as you can imagine I never used the Simple queries.
I got used to aggregates, but I must admit that Kilian present several suggestion that could could be interesting to implement it in order increase userbility of the aggregates.

Kilian, thanks for you long, clear and enlightening post.

Hello Kilian and all,
First, thank you all very much for taking the time to give us all this great feedback! This is truly an awesome community!  Thank you Kilian in particular, for the amazing effort of putting into writing such a complete description of the problem. 
Let me first share the good news: some of these issues have already been addressed! In particular, we made significant performance improvements to aggregates, we fixed the “Add Source” behavior, and we’re showing the original entity when using aliases. This in addition to other usability improvements we’ve been adding to aggregates, and that will be released next month.
However, I do understand that this doesn’t solve the issue of aggregates vs. simple queries. To address this, we’ll need a bit more time - we will need to dig deep into the problem, design and implement a solution, and bug you a bit more to have more in-depth feedback and real-world examples where the aggregates aren’t living up to your expectations. I can’t give you any timelines yet, but I assure you we are closely looking at this.
I’d like to stress once more how important your input was for us to understand the importance of this issue. So thank you again for your feedback, and keep it coming!
Hi Rodrigo,

Glad I could help a bit with ironing out this new, and on its own very useful, feature of the platform. It just saddens me that simple queries were removed at such a late stage of the beta 9 programme, so we hadn't much time to give you our concerns. Can you give us the reason for ultimately removing them? 

Best regards,

Currently as a manager of a major project of a major customer, i am thinking of making an upgrade to the latest version of the OutSystems platform.
However with this situation and with the opinions that I see here, I am very reluctant to go to the upgrade of plataform 9.
I prefer to wait for news before making a decision.
Best regards,
Hi Ivan,

The feedback on this thread is very good and pertinent. Our Engineering team is already addressing some of it and the aggregates feature will continue to be fine-tuned.

However, I can share with you that the feedback we have been receiving has been mostly positive. Here are some quotes we’ve been receiving from customers and the community, when we ask them about feedback on that particular feature:
  • "Aggregates are useful particularly when creating a complex query, it visually shows you the results of those queries preventing any possible query issues (like multiple join record problem). It is useful particularly on creating reports and dashboard charts."
  • "The Data Preview is something Great (5*)"
  • "Love the ease of putting together filtered and grouped data."
  • "We used aggregates to replace the advance queries."
  • "Aggregates help non proficient developers remove complexity when building common queries like Sums, Group By with immediate preview."
  • "I think this is a huge improvement to the platform and will allow more developers (not experienced in SQL) to show the information with nice visual elements (like graphs)."

OutSystems Platform 9 is packed with many other improvements. We have customers who have upgraded because they wanted to take advantage of the native mobile capabilities; others wanted to easily consume REST services; we have some customers who are integrating with DB2 databases; customers that are using the new database API to build their own database plugins for DBMSs we don’t support out of the box… You have many reasons to upgrade right now. ;-)
Ivan - I totally understand the concerns! Here's my take on it:

* Aggregates aren't perfect. Neither were Simple Queries or Advanced Queries.
* OutSystems has always been extraordinarily responsive to the forum posts, bug tickets, and back channel communications that I have had with them.
* Aggregates, some UI issues aside, are a HUGE improvement over Simple Queries on the whole. I can now do aggregate functions without the complexity and maintenance issues of an Advance Query and without the ugly code to do it in code after a Simple Query.
* The OS 9 upgrade contains a massive number of very attractive features (.NET 4.X and REST to start with!).

Bottom line: Aggregates may not be perfect, and there may be an adjustment period with them... but we are moving ahead with an upgrade to 9 as quickly as we can (we have a SINGLE application that is nearly 1,000,000 software units in 8!), and aggregates are the only thing that cannot be seen as a 100% improvement in version 9.

Justin James wrote:
we have a SINGLE application that is nearly 1,000,000 software units in 8!
 That's nothing: ours is > 1.4M :) But I agree with Justin, version 9 is the way forward.
Dear all,

This discussion, as I started it, is NOT about if version 9 is good or bad.
I know version 9 is a way forward, as expected by a vendor like Outsystems.

The issue here is whether an excellent functionality like Simple Queries should just be discontinued, when it served a perfectly good and large set of use cases (and in my opinion way better than aggregates serve now or probably will serve even after being improved in next version).

So, my first question still stands: "Is there any good reason why platform 9 can not have both Aggregates and Simple Queries alongside ?"
(and previous posts already answered the situations where Simple Queries are a better fit)

Pedro Moita.

That is a good question, and I think it boils down to a management decision. Simple Queries, even improved compared to platform 8, where present in platform 9 beta versions, alongside aggregates.
Hi all,

I´m also of the opinion of have Simple Queries and Agregates living side by side so after reading this post i created the idea so we can see the amount of developers feel the same way about this.
Because no one have doubts about that aggregates are very powerful but in lot of cases the Simple Queries do the same and are more appropriate.

Romeu Azevedo