Urgent Help on RecordList Inserting

  
I have an recordlist(could have 5million records)generated from split_string action. Examples of the record inside looks like 

123-ABC-12
2013-01-02
12
233-EWT-33
2013-01-02
24
873-DEF-92
2013-01-02
32
113-QWE-13
2013-01-02
2 

As you can see there is an pattern and every 3 records is actually one record of this new entity called Book and it contains 3 fields: num, date, qty

My question is how can I insert every 3 records into Book from the split_string action which contains the recordlist

All the method I tried are inserting way way to slow

Thanks

P.S. above is just an example, the pattern isn't 3 can but up to 12, but in the case where every 12 records in split_string's output = 1 record in the Book Entity, I just need to insert only the First Record, Third Record, and the Twelve Record inserted of the 12
 
My Really Poor Performance could takes hours to completely insert is here http://www.outsystems.com/forums/discussion/11134/reading-in-delimited-txt-files-extension/

Why I have 5 Million records from the split string?

Well Since outsystems only support Excel to RecordList in the link post I ask if its possible to have Tab Delimited Txt to RecordList, and I did it. However, the text files contains over 500K Records with 12 columns, so thats pretty much over 5 millions lines when you try to manipulate the string (we can see how I did it in the link)

Hopefully someone can help, I spent over 4 days just on this (uploading tab delimited) problem :(


Hi David,

What solution from that post you linked are you using? The one Justin James explained should be alot better than the one from your screenshot, since it is all done in memory instead of using the database for temporary storage.
The only interaction with the database that you should have is the 1 Insert per line.

So instead of replacing the line break by the delimiter, do like Justin sugested:
  • first do a split by line break
  • do a foreach over that result (should have the 500k lines separated)
  • then in each line split by the separator
  • do a foreach over that result (or access the positions you need by index, like String_Split.List[1].Text.Value) to fill your entity record
  • Use CreateBook to insert it in the database.

Regards,
João Rosado
Hi David,

Take a look at the CSV component on the Forge (you can specify the delimiter to TAB instead of comma).
It hasn't been updated in a while but it's worth a look. :)

Cheers,
Davide