Outsystems now iOS crashing while synching 1.000.000 records

Outsystems now iOS crashing while synching 1.000.000 records

  

Hi all,

I am trying to synch 1.000.000 products from my online database to my local database on my ipad. I only place the 4 relevant attributes of the products in the local database. My problem is that the outsystems now app crashes after a while synchronizing, sometimes this happens after approx. 2 minutes, and sometimes after approx 5, so that is pretty random. No errors are given, I just see the homescreen of the ipad when it crashes. My suspicion is that is crash is caused by a memory overload. 

Can someone confirm this? And are there any solutions for this problem?

When I test the synchronize function in the web emulator, everything works fine.

Hi, 

Have you tried to synch the local DB by various times, say 200.000 products each time? 

It can be also a problem of timeout. 

Regards

Graça Peixoto

Maria da Graça Peixoto wrote:

Hi, 

Have you tried to synch the local DB by various times, say 200.000 products each time? 

It can be also a problem of timeout. 

Regards

Graça Peixoto

Hi, thanks for your reply. I must admit that I have thought about processing it in batches. But I'm not sure how to implement this solution, since timers and processes are server actions and can therefore not be used for client actions. Do you know how to achieve this?


Hi,

There are limits of local storage, so you have to take that into account (think of 5Mb - 10-Mb)

http://dev-test.nemikor.com/web-storage/support-test/


Furthermore, localstorage is local for a reason, you don;t want to store everything there, so why do you need 1 million products?

Imho, you should think this over and only sync "used" products on localstorage and on demand certain categories of products. for example





J. wrote:

Hi,

There are limits of local storage, so you have to take that into account (think of 5Mb - 10-Mb)

http://dev-test.nemikor.com/web-storage/support-test/


Furthermore, localstorage is local for a reason, you don;t want to store everything there, so why do you need 1 million products?

Imho, you should think this over and only sync "used" products on localstorage and on demand certain categories of products. for example





Thanks for the reply. I looked at your test website, and found it hard to understand. When I run it in Safari on the Ipad it goes to 5000kb, but is outsystems now using the same localstorage as Safari? And is WebSql part of the local storage?

Anyway, I further tested the synchronization of the products as follows: I downloaded the first 500.000 products first to the Ipad. Then afterwards the second batch of 500.000 products, without deleting the first batch. This succeeded: 1.000.000 products correctly downloaded. This suggests that storage space is not the problem here. So my suspision still is that the amount of available memory is the issue. Do you agree?

Maybe automatically processing the products in batches is a solution, but how do I implement this? Since I can't use timers/processes in the client action...

p.s. for our use case it unfortunately really is neccessary to download all 1 million products to the local storage


1million is still plenty.


but you will have to do something with a status.

out of my head i would approach it little bit like this:


perhaps simply divide the products into groups (dummy groups).

for example groups based on the first letter of the product or whatever suits the proper distribution of equal sizes.

those groups have a status of "synced", "requestsync", "syncing".

I suggest, because this is so large, a manual sync would be in order.

by simpling pressing a button to check if there are groups to be synced.

when starting the app, you check if there is at least 1 thing to sync and redirect it to the sync-page...


and per group you will have a query that will fetxh, for example all records starting with A.

those are synced to local storage and give feedback.

next click on the button will fetch the products with B.

etc,.





J. answer is great. Before computers were constantly online, it was frequent to do syncs with the server-client model. There each row server row has a field saying when it was last updated.

The client knows when was the last update made for each table. Clint asks the next x row, and the server returns the x+n rows ordered chronologically (some rows may have the same timestamp). If the lines are successfully inserted, update client's lastupdate table to the biggest received DateTime and repeat for another batch until the answer is 0 rows.

This way, you don't have to change data structure on the server, and on the client, there is only one extra line of information.


Note: This doesn't work well for huge chunks of data, so if all your date enters the server with the same timestamp, you need to force some artificial pauses or do some other sequential numbering.

Hi Bas,

If your question is about "how" to do a partial "sync" in background (without the need of the user to do it), you can do it with Cordova, I think.

For example, take a look at this: https://developer.chrome.com/apps/alarms

Probably you can also do it with JavaScript, as the Mobile App is a "single page" app...

Cheers,
Eduardo Jauch

Thank you for your reactions, I will look into the last two options; the groups with a sync-status and the cordova/alarm. The last option seems most suitable for my situation, because it is preferable to keep it as simple as possible for the end user.