164
Views
7
Comments
How to keep data in memory and only save in bulk, to reduce database/server calls
Application Type
Reactive

Hello,

I'll divide this in a "long exposition" and a "short exposition"
Thanks in advance!

LONG EXPOSITION:

I'm implementing an audit system (in a Reactive application) that registers every interaction that is established with the data of the app, even its consulting - which will result in a lot of info getting created every time the user interacts with the application, that will hinder performance.

To try to reduce the calls made to the server as it is saved, I'm searching for a solution that allows me to hold on to the audit data in memory and only submit it to the database, in bulk, after a certain amount of time OR interactions made the user.

I had a few ideas in mind on how to go about it, but so far they're flawed.

The latest I'm considering is saving that data on a local variable on the page the user is navigating and only commit it to the database when:

1) he navigates to another window of the app (by clicking on a link or by "going back" to the previous page via the browser)
2) he closes the browser
3) his session times out

2) and 3) seem to be covered on these threads:
https://www.outsystems.com/forums/discussion/58273/reactive-web-app-warn-before-leaving/
https://www.outsystems.com/forums/discussion/71078/reactive-web-run-action-on-logout-events-close-browser-timeouts-etc/
[it alerts to browser-specific behavior, though]

However, my problem is with 1) - how to hang on to that local variable data if the user navigates through-out the app. Even if I were to do a strange measure like sending that data from page to page with input variables, that would not cover the case where the user backtracks with the help of the browser option.
Also, to save this type of data as a Client variable is not doable, both because of size/type of data and the preservation of the app's performance itself.

If anyone has a suggestion for case 1); or if anyone has faced something similar in the past and came up with a different efficient approach, I would be very thankful to hear your tips!

____________________________________________________________

SHORT EXPOSITION: I'm trying to keep data in memory as the user navigates the app, and only commit it to the database, in bulk, at certain times, to reduce calls to the server.

Thank you!

2018-06-05 16-54-03
Maria da Graça Peixoto

Hi!

I understand your problem but I don't think that keep the data in the screen is a good idea. 

Letting the data "alone with the user" is always a invitation to some one try to tamper it, what would ruin the very purpose of your work. 

Perhaps reducing the data to the minimal  and use info already in the server whenever it is possible, but sending the info immediately to the server would be the safer way. 

Just some ideas.

Graça

UserImage.jpg
João Francisco

Thank you very much for taking the time to answer, Maria

I agree, it's not the best approach. I'm considering it given that the auditing mechanism will be very exhaustive and the data that it produces is not a source of worry in terms of the app's security/integrity, since it will just record who accessed what at which time.

I will take your opinion into consideration! :)

Thank you, once more,

João


2021-09-06 15-09-53
Dorine Boudry
 
MVP

Hi Joao,

don't have the anwser for you, but here are some thoughts :


# you are looking specifically for a solution for interactions of the end user with the UI, right ?  Any server requests or api calls etc. don't fall under the problem you are having ?


# can you elaborate on what exactly you want to log, is it more tied to the Outsystems structure / logic / bespoke / specific side of your screens, or more to the html / javascript / dom event / generic / technical side.

if it is at a low html / javascript / generic / ... level, maybe you can look at integrating already existing tools to do this, that are not specially geared at Outsystems applications.  

If it is strongly tied to the Outsystems platform or to your specific business needs, then that might not be an option / might be overkill.


# not sure if using client variables would pose a giant performance problem, if you only use it whenever somebody enters or leaves a scope (like a screen, but also maybe at block level)


#Can you share more about what type of data and the amount you are trying to keep track of ?  Is it all moves done by the user, every single click of mouseover, is it all changes to the local data, is it all executions of client actions , ... ?


Dorine

UserImage.jpg
João Francisco

Thank you very much for giving me all these points, Dorine!

I think this paragraph below will address everything you asked :)

I'm trying to log the "creation"/"edit"/"delete"/"access to" data on the app. The "access to" part is the one that will be the most recurrent, which made me consider a way to not constantly connect to the database. So, in this case, I'm trying to save audit data (username, affected element, date of interaction, type of interaction) at each interaction. The fact that, in order to save it, I'd need a structure, removes the possibility of using a Client variable - even if I were to toy with the idea of converting it into a text, via JSON, the JSON would be too large a string to hold.

Thank you very much, I really appreciate it

João

2021-10-09 07-57-44
Stefan Weber
 
MVP

Hi Joao,

depending on your development environment i would suggest the following:

If you have an on-premise installation and access to the SQL Server environment you could make use the SQL Server Audit feature for server and databases and database objects.  See here  https://learn.microsoft.com/en-us/sql/relational-databases/security/auditing/sql-server-audit-database-engine?view=sql-server-ver16 and here https://learn.microsoft.com/en-us/sql/relational-databases/security/auditing/create-a-server-audit-and-database-audit-specification?view=sql-server-ver16

If you have a cloud pipeline you cannot make use of the above. Now in my opinion an audit trail should be 100% accurate. I doubt that you will find any decoupled approach - like the one you describe yourself - that will guarantee that the audit trail is a full representation of the users actions. To make it accurate writing the audit entry must done within the same transaction of the auditable action.

For create, update and delete you would directly add the audit create actions in the flow. This indeed would involve two database roundtrips, but i guess for changes that is ok.

Selecting data can indeed be a problem as you may need to write an audit entry for every resulting record. I never had the requirement to audit list results, but just accessing detail data, so the following is just a way i can think of:

Create server actions to retrieve data using advanced sql. An advanced sql could look like the following

WITH QUERY_CTE
AS
(
  SELECT {DataTable}.[Id],{DataTable}.[Name],{DataTable}.[Other]
  FROM {DataTable}
  WHERE (@SearchTerm = '' OR {DataTable}.[NAME] LIKE '%' + @SearchTerm + '%'
)

INSERT INTO {AuditTrail} (Id,UserId,Table,Name)
SELECT
  NEWID() AS Id,
  @UserId AS UserId,
  'DataTable' AS Table,
  QUERY_CTE.[Name] AS Name
FROM QUERY_CTE

SELECT
  QUERY_CTE.[Id],
  QUERY_CTE.[Name],
  QUERY_CTE.[Other]
FROM QUERY_CTE

Please note that i did not test the above SQL.

Another option is to add an event store to your stack. Like https://www.eventstore.com/

Using an event store you would query your data with an aggregate and then push the result with additional audit data to event store.

Stefan




UserImage.jpg
João Francisco

Hello, Stefan

Thank you so much for the detail you put into your answer, and I'm sorry for how long it took me to respond.
I agree with your approach of a 100% accurate audit trail, so we're going to go with a way of thinking that most resembles that but still looks out for some of the originals concerns I had: wherever any "hard" operations occur (create, edit, delete), these will be accompanied with an immediate audit creation. As for the "consulting" audits, we'll store those in a local variable on the page and only commit them to the database when the page is destroyed, that way economizing some server calls.

I did not know the features you mentioned in your comment so I'll also look into them, since they might be useful to me.

Thank you, once more, for all your help

João Francisco

2021-10-09 07-57-44
Stefan Weber
 
MVP

A well asked question deserves an expressive answer :-)

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.