Load VERY LARGE JSON file (700Mb) into Database
Question

I have a large JSON file (700 Mb) that was created as an extract of a database. I now want to "import" into OutSystem entities.

I successfully created the Structures based on the JSON (super easy 2-clicks)!

Then, I created a very small version of the JSON file with only 2 records in it and build the loader by using BinaryDataToText and then JSON Deserializer ... BUT ... when I add the 700M file as a Resource, the app will NOT deploy with this error below.

Is there a better way to load such large files into an OutSystems DB table?

------------------------------------------------

Internal Error

An error occurred in task 'Load OML':

Index was outside the bounds of the array.

Details : System.IndexOutOfRangeException: Index was outside the bounds of the array.

   at Noemax.FastInfoset.XmlFastInfosetWriter.#co.#jy(String , Byte[] , Int32 ) .....

So if I understand correctly the issue lies with the fact that you can't deploy the application when you add the 700MB file as resource. The import works as expected when you use the smaller file. 

I would then add an upload functionality. Create an application that can accept the file and then start the upload process from there. 

By the way, I don't understand why you would need the BinaryDataToText action. The JSON file is already Text so why would you want to convert it?

Greetings,

Vincent

I will try that and see if the upload times out or not. 

Great question, for some reason the json file when uploaded as a resource, OutSystems evidently stores the resource as a binary. Thus, I have to convert the Resource from Binary to Text. Good news, that part works.

Given the file size, I think I will have to store the JSON file as Binary in the DB as well.

To complement Vincent answer, try to upload the file to the database and create a timer to accomplish the import, otherwise, you will see a timeout issue.

Probably you will also see a timeout if you try to upload that big file. You can do this to change the upload file limit:

https://success.outsystems.com/Support/Enterprise_Customers/Maintenance_and_Operations/Altering_file_upload_limits_in_OutSystems_apps

@luis - hmmm ... I am running v11 and in a personal env. That link shows how to change the Factory but it looks like maybe OS9 or OS10. How would I change the limits on OS11 in a personal environment

Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.