ListDuplicate breaks Simple Queries Optimization

ListDuplicate breaks Simple Queries Optimization

  
Hi!

I'm writing this message to see if someone else experienced this issue.

We have a File Entity that has:
Filename (Text)
UpdateDate (DateTime)
Content (BinaryData)
We have a Web Screen that list File records and that allows the user to:
  • select all files and delete them
  • download files
As the web screen was taking a considerable time to load and had a considerable page size (1.5 Mb) we started to check what could be wrong.

First we noticed that Content was being used on the download screen action without first being fetched from the DB. That is problematic because that way the content need to be stored on the viewstate so we change it to fetch content just before the download action.

As Content was not being used anywhere else we were condidrent that we solved the problem, but we didn't, the page was still slow and the page size the same.

To try to figure out we did a test using an Advance Query that only gets Filename and UpdateDate and with this new query the page size was 71 Kb and the load time much better. This was what we expected but the result didn't make any sense, Simple queries only retreive what will be used so why it was retreiving the Content?

After some tests we figured out the culprit, the ListDuplicate inside the Delete All Selected.

So it seems that, even if the the ListRecords doesn't used the Content attribute, because of the use of ListDuplicate action all attributes are fetched.

After we changed the logic to not to used ListDuplicate while using a simple query the page load time improved (a lot) and the size was the same as when using the Advance Query (71 Kb)

OS version: Version 8.0.1.25





Hi Ricardo,

you should isolate binary data in another entity. Are you deleting your files one by one? Just use an advanced query and do a bulk delete.
Hi João,

Thank you for your reply, I agree with you on with the idea of having the BinaryData on another Entity but my question was more about the fact that the ListDuplicate "breaks" the optimization. Did you ever notice that?
Well, it's kinda obvious. listduplicate does what it has to do. Duplicates the list. It does it 100% otherwise you get nasty sideeffects... Imho its correct design
Hi ,

I don't agree with you, if I'm duplicating the ListRecords that only has a subset of values why it's duplicating everything?
Hi,

It's duplicating everything, because you are at design-time.
The structure  has to be known, because if you assign attributes, alter attributes.

Since optimization is being done runtime, if you want what you want, is also optimzing every (sub)action
that *might* use the list-duplicate.
which can make life much more complicated than you actually want.


The keyword here is, optimization is being done "compiletime" and *NOT* design-time.



I think optimizations are done on compile time and not on runtime, and so the platform knows everything that is used by the webscreen and by its actions. 

So, if the attributes are being used they need to be fetch, if they aren't then ListDuplicate is duplicating something that will not be used. Resuming the current way is, besides breaking the optimitations, copying to memory unnecessary data.

Again, maybe there is a good reason but I'm not seeing it.
I understand your reasoning, but the logic is getting too complex to gain actual advantages over it.

ScreenAction1: ListDuplicate, call action 1, calls action2, shows name, surname, lastname, accountnumber
Action1 : Loops through the list, duplicates odd number to a new list , calls action 3
Action2: updates the list and set the lastupdateby
Action3: loops through list, validates street

ScreenAction2: calls action 3, shows name, telephonenumber, birthdate

So, designtime you have everything.
before compiletime, outsystems needs to pass through everything recursively to actually know the complete subset of items which are actually used or not.
This means evaluate all if-conditions, switches and assigns in every action.
This is only in 1 espaqce, what if you have those actions and/or webblocks as public and you have multiple-espaces..


Hence the reason, don't overcomplicate things. just optimize a direct aggegrate from a webscreen.
Plain and simple.


What do you mean by "optimize a direct aggegrate from a webscreen" I'm still on version 8 so I don't have Aggregates but I would like to understand your suggestion.

PS - I had already found a way of doing what we need without using the ListDuplicate, when I created the post. This is just bothering me because I was promoting the use of simple queries because they are automatically optimized and also because scaffolding "automatically and silently" makes the pages slower, as it uses the ListDuplicate for the "Delete all selected" option (when applicable).
oops, aggregate is the new simple-query in 9 :)


I'm sorry Joost, but there is no comparison between aggregates and simple queries *wink*

Statler & Waldorf and the amazing flappy tweedles! wrote:
oops, aggregate is the new simple-query in 9 :)

 
 
 
ok ok,

the simple query is dead, long live the aggregrate!

Solution
Hi Ricardo,

Long time no see :). Still at Logitech?

Anyway, with regards to the optimizations, as J. (where's Statler and Waldorf gone? :)) explained, the platform cannot really know what you're doing with the duplicated list once it's been duplicated. It can decide that the original list is only used in a way that allows the optimizer to limit the data retrieved, but it would be an optimizing nightmare to also trace what happens with the duplicated list. In these cases, it's a good thing to realize when and how the platform optimizes, and prevent programming structures that cause the optimizer to throw the towel in, so to speak.


Solution
Hi Kilian!

Yep, still working at Logitech :)

That's a very good example. Even if not impossible it will be a complicated trace of used attributes.

Anyway, at least Scaffolding should be optimizer friendly and not use the ListDuplicate on the "Remove Records" action. When I have more time I'll try to suggest something.

Regards!

Well, finding a way was stronger than me.

Here it goes:

Before:



After:


I'm using a Integer Structure List to store the "currentRowNumber"