Best Approach

  

Good evening Gurus,


I need to know the Best approach to do the following:

I wanna build a platform that do some processing on a Database uploaded by a user, then the user can download it back again (After the platform process it).

I don't wanna keep a copy of the DB on server (For Server space management).

The DB will have a certain structure, entity attributes are pre-defined from platform side.

Any ideas what are the alternatives to build this ?

You could store it entirely in structures if you do not want to write it to an entity in OutSystems.

So pull it in, assign to a list of structure records, manipulate to a new list of structures and then export again.

Is it coming in in a predictable format?

Jordan Welch wrote:

You could store it entirely in structures if you do not want to write it to an entity in OutSystems.

So pull it in, assign to a list of structure records, manipulate to a new list of structures and then export again.

Is it coming in in a predictable format?


I started to implement it this way, but i'm facing one issue -may be coz of my lack of experience in OS - that i used to use aggregates to sort and manage lists. that was available for server entities but not structures. Any work around ?

For your question, let's assume for simplicity sake YES it shall be in predictable format. Maybe i add another post for that later :)

So if you take a look at outsystems System actions (linked here) you'll see there are a whole bunch of actions for manipulating Lists. You can use these on lists of structures. You should be able to achieve most of the same actions as an aggregate with a bit of ingenuity.

Thanks Jordan, I'd like to share my progress and raised concern till the moment with you.

  1. I created a structure entity with all attributes that the platform is expecting in the uploaded excel file.
  2. I added a LOCAL Variable at the screen where Upload shall take place and a Table Records to show the Uploaded content, Upload widget to select the file to Upload, and a button to assign the "Excel to List" to the Local Variable

My concern is the Upload action-Linked to the Upload Button- is using Submit action to assign the content to the Local Variable. Screen re-loads but the Local Variable still holds the new Uploaded content.

How this happens ?!!, As far as i know the life time of the Local Variable expires after page reloads. any justification ? This action is actually what i want but i'm not confident that it'll keep acting same way when i keep developing the application.

Hi Sayd,

To be honest, this sounds like a pretty rediculous idea. OutSystems is a business tool meant for dealing with large sets of data, data entry into the database, and data processing (using the database). From what you want, it seems that OutSystems is not the right tool for you, even if you could make use of it. It's like saying you want people to drive a Porsche, but due to petrol management reasons, they can't actually use petrol. Doesn't make sense.

Kilian Hekhuis wrote:

Hi Sayd,

To be honest, this sounds like a pretty rediculous idea. OutSystems is a business tool meant for dealing with large sets of data, data entry into the database, and data processing (using the database). From what you want, it seems that OutSystems is not the right tool for you, even if you could make use of it. It's like saying you want people to drive a Porsche, but due to petrol management reasons, they can't actually use petrol. Doesn't make sense.

Hi Kilian,

I believe that Outsystems is the right tool definitely, otherwise i wouldn't be posting here.

Let me re-introduce the idea i work on, that might shed some light on what i need and please keep remembering that i submitted this post originally seeking advice for the best approach.

I plan to build an online platform that a user can upload to a Database to be processed in someway (This is where my know-how resides), and then Download it back again. That's all. I know that we can store all Uploaded & Processed DB and link it to userId, But i've no business justification at the moment to do so.

You can say that i take the advantage of processing power and easy programming of OS; Do you have a better approach to suggest ?


So the page Variable will reset if you reload the page outside of that submit, but not just on the submit. That would be the biggest argument for creating an entity to store it, even if you then clear out the Database at the end of the processing. 

I think you're definitely making more work for yourself using entirely structures, but I don't think the idea is too ridiculous. OutSystems may be more powerful than what you're using it for, but the glory is you can use it for whatever you like and it can still work in some fairly unusual ways.  

Sayd,


What you want is: User has a bunch of data that needs to be processed (excel file), you want to do it in memory and download the processed results. Is that what you wants to do?

Well, all local variables will be put in the View State of the page. So, if you have a huge set of data, you are in bad waters regarding performance (first thing).

Second thing, the minute you navigate away, you looses all your "memory information". So, you just can't keep your information in memory. You could think of passing all this info to the new page, but again, or you will have a huge performance penalty, or you will not be able to do it at all, as there is a limit to the size of input parameters (I think).

So, if the thing is "simple", you submit the file, convert, process, create a new file with results, and download, or you will not be able to do what you want.

Cheers.

I'd like to thank all of you for sharing your thoughts about what i'm about to test.

Now it's the time to drive the Porsche all the way down the hill using gravity force not the Petrol Pedal :)

And to what Eduardo said: OutSystems does not let you pass a list as an input parameter, so it will have to all be on one screen unless you save it somewhere.


Good luck

Jordan Welch wrote:

And to what Eduardo said: OutSystems does not let you pass a list as an input parameter, so it will have to all be on one screen unless you save it somewhere.


Good luck

You can serialize it :)
But the size would be huge... (and probably the browser/server would deny it).


Eduardo Jauch wrote:

Jordan Welch wrote:

And to what Eduardo said: OutSystems does not let you pass a list as an input parameter, so it will have to all be on one screen unless you save it somewhere.


Good luck

You can serialize it :)
But the size would be huge... (and probably the browser/server would deny it).


Thanks you actually may have solved a problem I was having with passing a smaller list into a page... Will have to keep this in my back pocket..


Jordan Welch wrote:

And to what Eduardo said: OutSystems does not let you pass a list as an input parameter, so it will have to all be on one screen unless you save it somewhere.


Good luck


Yes Jordan, I kept all business in One Screen. I think it's a good chance to learn some new stuff.

Sayd,

Apart from me still thinking you have a strange business model, you really shouldn't want to keep all data out of the database. What you could do is store the data temporarily, process it, and let the user download the processed data, then after a set time clean up that user's data. The "disadvantage" (from your point of view) of this would be that you need to reserve some database space (temporarily), but the advantages are huge: you can process using timers (which allows for bigger datasets and/or more complex calculations, avoiding page timeouts that you're bound to get when processing such data in a Screen Action, and allows you to schedule processing), and have the user download multiple times if wanted (for a set time). You could even monitor the user's data and allow some users a bigger quota (or have them pay for more space).

tl;dr: even if you don't want to store user data permanently in your database, storing it temporarily has huge benefits*

*Drive the Porsche on petrol, but only lend it when it's needed :)


What is the expected size of the data? how many users do you expect to have (total, concurrent etc)?

Using the view state is a reasonable solution for small amounts of data but you need to remember that everything in the view state needs to be passed through to the server on every refresh, post and screen action and that takes bandwidth plus uses memory on the web server. I remember reading somewhere that the web server can temporally use anywhere up to 3 times the post size in memory (not an Outsystems thing, just standard webserver memory architecture). So take your maximum post size multiply by the number of concurrent users and then multiply by 3 and see what you come back with. 

Dropping the data into an entity is one valid option, and deleting it after Either on the download or a timer. You could either do it as one record with a binary or JSON object or do it as separate records depending on how much you wanted to use the power of SQL in your processing.  Just watch out for database size blowout and fragmentation with a lot of insert delete operations or large fields. If your worried about database performance then just get your indexes right (too many will kill your delete and inserts, wrong ones will kill your reads). If you are only ever doing one user at a time consider doing a truncate table instead of deletes as that's significantly faster but will kill ALL records in the entity regardless of tenant, Just don't do a record by record delete in a for each loop do it via an advanced SQL DELETE query with a filter. 

Other option is to link to an external service, eg shared filesystem, Azure Redis cache etc but then you don't have the aggregate (ie SQL) features.

Kilian Hekhuis wrote:

Sayd,

Apart from me still thinking you have a strange business model, you really shouldn't want to keep all data out of the database. What you could do is store the data temporarily, process it, and let the user download the processed data, then after a set time clean up that user's data. The "disadvantage" (from your point of view) of this would be that you need to reserve some database space (temporarily), but the advantages are huge: you can process using timers (which allows for bigger datasets and/or more complex calculations, avoiding page timeouts that you're bound to get when processing such data in a Screen Action, and allows you to schedule processing), and have the user download multiple times if wanted (for a set time). You could even monitor the user's data and allow some users a bigger quota (or have them pay for more space).

tl;dr: even if you don't want to store user data permanently in your database, storing it temporarily has huge benefits*

*Drive the Porsche on petrol, but only lend it when it's needed :)



Kilian,

could you introduce me to good resources to learn about "process using timers" solution ? my search didn't help for that topic. I usually use screen actions to process data.