We are getting errors when syncing binary data from Local Storage to our server, and it seems like the local device is running out of memory trying to use a StringBuilder. I suspect this is because it is trying to read the BinaryData column of the table we are trying to sync, and there is a video in there.

We will get an error similar to the following any time we try to add a video of around more than 5 seconds.

Does anyone know the best way to sync binary data to the server? Should we carry on doing it as a sync like this, or should we look to maybe try and create a file locally and send it up via a REST API or something else entirely? After a bit of advice here :)

Currently we are doing a select of the rows from local storage, sending these to the server as a list, and then looping through that list and using the Create[Entity] action to create that row.


Here is the error logged in OutSystems:

Error Detail
Back to Log
Id:
e5f82d68-d07b-403b-a0c6-347bcc7a4b6f
Time of Log:
17/04/2019 12:57:00
eSpace:
Service360GO
Tenant:
User:
Session Id:
No session
Server:
E3PV4-NP4SWT
Module:
OSCrashHandler
Message:
A fatal error has occurred. Please contact OutSystems support: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 76MB until OOM, max allowed footprint 146390472, growth limit 201326592
Environment Information    eSpaceVer: 17 (Id=1389, PubId=1469, CompiledWith=10.0.828.0)
RequestUrl: [https://servername.co.uk]/[ModuleName]/moduleservices/log?clientTimeInMillis=1555506040607 (Method: POST)
AppDomain: /LM/W3SVC/1/ROOT/Service360GO-101-131999640738180342
FilePath: d:\OutSystems\Platform Server\running\[ModuleName].458188794\
ClientIp: 10.12.242.167 X-Forwarded-For: 80.194.86.146
DeviceUUID: 6ba624d2e1418fba
Locale: en-US
DateFormat: dd/MM/yyyy
PID: 4412 ('w3wp', Started='4/8/2019 8:53:14 AM', Priv=1026Mb, Virt=10602Mb)
TID: 51
Thread Name: 
.NET: 4.0.30319.42000
DeviceModel: SM-A530F
DeviceUUID: 6ba624d2e1418fba
OperatingSystem: Android 9
NativeShell: 4.1.0
AppVersionCode: 1
AppVersionName: 0.2
NetworkStatus: Online
NetworkType: WIFI
Cordova: 6.4.0
Client-Side Log
Stack:
java.lang.OutOfMemoryError: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 76MB until OOM, max allowed footprint 146390472, growth limit 201326592
at java.util.Arrays.copyOf(Arrays.java:3260)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:660)
at java.lang.StringBuilder.append(StringBuilder.java:203)
at org.json.JSONStringer.string(JSONStringer.java:344)
at org.json.JSONStringer.value(JSONStringer.java:252)
at org.json.JSONObject.writeTo(JSONObject.java:723)
at org.json.JSONStringer.value(JSONStringer.java:237)
at org.json.JSONArray.writeTo(JSONArray.java:613)
at org.json.JSONStringer.value(JSONStringer.java:233)
at org.json.JSONObject.writeTo(JSONObject.java:723)
at org.json.JSONStringer.value(JSONStringer.java:237)
at org.json.JSONObject.writeTo(JSONObject.java:723)
at org.json.JSONStringer.value(JSONStringer.java:237)
at org.json.JSONArray.writeTo(JSONArray.java:613)
at org.json.JSONArray.toString(JSONArray.java:585)
at org.apache.cordova.PluginResult.(PluginResult.java:49)
at org.apache.cordova.CallbackContext.success(CallbackContext.java:88)
at io.sqlc.SQLiteConnectorDatabase.executeSqlBatch(SQLiteConnectorDatabase.java:148)
at io.sqlc.SQLitePlugin$DBRunner.run(SQLitePlugin.java:362)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)

So, you're saving the video to a LocalStorage entity, and then syncing that to the server using the sync framework?

Would it be correct to assume that the video data is captured on the device?

I think, given the data size we're talking about with video, I would be inclined to look at a separate process for uploading videos, rather than using LocalStorage and sync. 

That's just my gut feeling, based on the order of magnitude of the file size, when compared to simple picture data or other binary files (audio, for example).

G. Andrew Duthie wrote:

So, you're saving the video to a LocalStorage entity, and then syncing that to the server using the sync framework?

Would it be correct to assume that the video data is captured on the device?

I think, given the data size we're talking about with video, I would be inclined to look at a separate process for uploading videos, rather than using LocalStorage and sync. 

That's just my gut feeling, based on the order of magnitude of the file size, when compared to simple picture data or other binary files (audio, for example).

Thank you for the reply, what sort of separate process would you think would work? I'm completely in the dark about where to look.

Russell Codd wrote:

Thank you for the reply, what sort of separate process would you think would work? I'm completely in the dark about where to look.

I don't really have specific recommendations to offer, beyond possibly creating a screen in your app that would allow manual upload of the video files once they're recorded, or keeping the video files separate from the rest of the schema, and using the mobile sync framework to periodically sync the files with the server.

The Offline section of the mobile app documentation is a good place to start in terms of the patterns for syncing offline data: https://success.outsystems.com/Documentation/11/Developing_an_Application/Use_Data/Offline