16
Views
1
Comments
OOM while syncing binary from server entity to local entity
Question

Hi,

I'd like to sync my pdf files and images on the device's local database from server entities
I'm following OutSystems' best practice for creating the sync process.

  1. The first step is getting document from the server database in loop (1 at a time)
  2. CreateUpdateAll action for creating records on the local database.

Everything work till sync is not broken. As we are porting around 500+ documents (each document is well within 1 MB only), there is plenty of opportunity that network will fluctuate. As soon as network fluctuate, sync breaks and then when next time it starts we get following error.

Please note, everything is fine till network breaks but after that when network is back and application is online. Respective table never sync and throws OOM as show below:

Error Log:

2020-08-23 00:27:17SimplifyModule (Simplify)A fatal error has occurred. Please contact OutSystems support: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 55MB until OOM, max allowed footprint 168324424, growth limit 201326592OSCrashHandler

2020-08-23 00:16:51SimplifyModule (Simplify)A fatal error has occurred. Please contact OutSystems support: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 55MB until OOM, max allowed footprint 167972272, growth limit 201326592OSCrashHandler

2020-08-23 00:12:25SimplifyModule (Simplify)A fatal error has occurred. Please contact OutSystems support: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 55MB until OOM, max allowed footprint 168811872, growth limit 201326592OSCrashHandler

2020-08-23 00:08:59SimplifyModule (Simplify)A fatal error has occurred. Please contact OutSystems support: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 54MB until OOM, max allowed footprint 169758568, growth limit 201326592OSCrashHandler


During this time i see following in General Log:

2020-08-23 00:27:24SimplifyModule(Simplify)Invalid extension  while fetching MIME typeOSCache 2020-08-23 00:27:24SimplifyModule(Simplify)Cache was successfully serializedOSCache 2020-08-23 00:27:02SimplifyModule(Simplify)Executing ServerDataSyncOfflineDataSync 2020-08-23 00:26:00SimplifyModule(Simplify)Executing ServerDataSyncOfflineDataSync 2020-08-23 00:25:31SimplifyModule(Simplify)Manifest serialized with successOSCache 2020-08-23 00:25:31SimplifyModule(Simplify)Cache was successfully serializedOSCache 2020-08-23 00:25:28SimplifyModule(Simplify)switchToVersion version: AXwrX943wNN+fKHrzEmBtAOSCache 2020-08-23 00:25:25SimplifyModule(Simplify)Caching operation was skipped: version AXwrX943wNN+fKHrzEmBtA is already available in cacheOSCache 2020-08-23 00:25:25SimplifyModule(Simplify)startCaching started for version: AXwrX943wNN+fKHrzEmBtAOSCache 2020-08-23 00:25:23SimplifyModule(Simplify)Invalid extension  while fetching MIME typeOSCache 2020-08-23 00:25:23SimplifyModule(Simplify)Cache was successfully serializedOSCache


Details of Error shows:

Thread Name: .NET: 4.0.30319.42000 DeviceModel: Lenovo TB-X304L DeviceUUID: ebad35208f01cc3d OperatingSystem: Android 8.1.0 NativeShell: 8.3.3 AppVersionCode: 27 AppVersionName: 0.28 NetworkStatus: Online NetworkType: WIFI Cordova: 8.1.0 Client-Side Log Stack java.lang.OutOfMemoryError: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 55MB until OOM, max allowed footprint 168324424, growth limit 201326592 at java.util.Arrays.copyOf(Arrays.java:3260) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:660) at java.lang.StringBuilder.append(StringBuilder.java:203) at org.json.JSONStringer.string(JSONStringer.java:344) at org.json.JSONStringer.value(JSONStringer.java:252) at org.json.JSONObject.writeTo(JSONObject.java:723) at org.json.JSONStringer.value(JSONStringer.java:237) at org.json.JSONArray.writeTo(JSONArray.java:613) at org.json.JSONStringer.value(JSONStringer.java:233) at org.json.JSONObject.writeTo(JSONObject.java:723) at org.json.JSONStringer.value(JSONStringer.java:237) at org.json.JSONObject.writeTo(JSONObject.java:723) at org.json.JSONStringer.value(JSONStringer.java:237) at org.json.JSONArray.writeTo(JSONArray.java:613) at org.json.JSONArray.toString(JSONArray.java:585) at org.apache.cordova.PluginResult.<init>(PluginResult.java:49) at org.apache.cordova.CallbackContext.success(CallbackContext.java:88) at io.sqlc.SQLiteConnectorDatabase.executeSqlBatch(SQLiteConnectorDatabase.java:176) at io.sqlc.SQLitePlugin$DBRunner.run(SQLitePlugin.java:357) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636) at java.lang.Thread.run(Thread.java:764)

Hi Sachin,

It seems your overloading the server. The proces you are using isn't resilient for network interruptions as your already noticed. To solve your problem you need a more advanced synchronization method. I don't know how you data is modeled but I would suggest a proces like this;

1. Create a method that the client can call to list all documents that need to be synced (use some synckey for this so the server can determine which documents the client already has, which it doesn't has and which need to be updated or removed.

2. Use this list to retrieve all documents one by one. And after you got one (1) document from the server you need to store it directly in your local database and update the synckey so that should the connection be severed the client and the server can correctly determine which documents needs te be synced.

I recently created such a sync method (although with a lot smaller data so I can retrieve data in bulk) and if you just make sure you sync in the smallest chunk possible and keep track of the proces you can restart it whenever you need to. 

Greetings,

Vincent