24
Views
4
Comments
[Formula] Formula Version upgrade
Question
formula
Service icon
Forge asset by Caio Santana Magalhães
Application Type
Service

Hello,
I have a question for the developer or anyone that did this in the past.
In my company we have what i believe is Version 3.0.4 of Formula. I can see a major update in most recent version. Is the swap a easy change or what should we account for.

Thank you.

2019-10-27 01-32-56
Caio Santana Magalhães
 
MVP

Hi, Ruben!

I may be biased since I'm the author of Formula, but here's my point of view.

If you're already using Formula 3.x.x in production and it's working well and stable, maybe upgrading just for the sake of upgrading might not be really useful as it won't improve things much, performance-wise.

With that said, if you intend to:

  • Refactor an existing project that uses Formula, or
  • Start a new project from scratch

Then you might be able to leverage the new logical patterns of the newer version of Formula, which is way more optimized for repeatedly evaluating expressions for a large number of records.

I intend to release a new version of Formula (currently published as version 5.x.x flagged as 'Under Development', use it at your own risk for now!) with batch-processing capabilities, as I am working on a project that needs to parse millions of records very quickly. Eventually I'd like to add more operators such as IN, CONTAINS, etc. Feedback wanted here.

UserImage.jpg
Ruben Magalhães

Thanks for the feedback @Caio Santana Magalhães 

Its not yet in production, but in pre-production.

What happens is that we have some cases where we have to evaluate expressions in a cylce, of dozens or even hundreds of times, its configured by the user. I noticed that the extension is kinda slow, so i saw your new version should improve it a little bit. Already tried the new version, it performs a little bit better but its not enough (maybe reduced 4-5s max in my tests).

2019-10-27 01-32-56
Caio Santana Magalhães
 
MVP


Hi @Ruben Magalhães, here are a few tips that I learned that will hopefully help you guys (and anyone else who reads this).

  1. Turn off Activity Logging. Open the extension in Service Center, go to the "Operation" tab, then uncheck "Activity Logging" and apply changes. This turns off extension call logs that add considerable overhead when the extension action is called frequently or in a loop. (I usually do this for all of my extensions as we never had a reason to check extension logs anyways.)
  2. Avoid passing large lists to public actions via input. Lists are usually passed by reference, which causes zero overhead regardless of their size. The one exception is when you pass your list to a public action from another module: in this case, the platform forcefully deep-copies the list before passing it to the public action. This may cause unintended high memory consumption and processing, especially if inside a loop (potentially becoming an O(n²) problem).
    • To work around this limitation, I've included the action Formula_Batch_AddFromRecordList, allowing me to pass a large record list wrapped with ToObject() to my formula in a single call.
  3. Calling XIF actions is expensive. Even after disabling Activity Logging, there's still some platform telemetry overhead that happens under the hood. Normally, it's not a big deal as it adds <1ms per XIF action call. However, if you need to invoke a XIF action millions of times in a loop, this small overhead adds up considerably and becomes several minutes.
    • The solution to this is to make the least amount of XIF action calls as possible. Hence the reason why I added a set of Formula_Batch_<...> actions to the component, so that I can load and evaluate a very large list (1M+ records) with very few action calls. This alone improved my benchmark from several minutes down to just 40 seconds.
  4. Performance is worse in non-production environments. This might sound obvious but it's always good to remember that non-production environments run your application in debug-mode, which worsens the aforementioned XIF action call overhead. That is to say that if your logic takes 3 minutes to run in your DEV environment, the same logic might take 2 minutes in PROD, even if both environments have the same server specs.
  5. Benchmarking helps a lot. Try measuring each step of your logic and identifying bottlenecks. That's what helped me figure out the right way of evaluating millions of records with Flee in OutSystems under a short timespan. I ran so many benchmarks that I decided to publish a tool named Benchmark Dashboard, hopefully this helps!
UserImage.jpg
Ruben Magalhães

Thank you! 
I will try and implement some of the suggested tips.

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.