Skip to Content (Press Enter)
UserImage.jpg
Ronald Grobler
51Views
4Comments
Importing large quantities of data
Question
Hi,

Please can someone give me some advice on how to process files faster.

I am having problems importing data.
The current file I am having problems with is an excel document has 190 columns of data, I removed all but 10 rows of data.
First, I convert the data into a record list. This is quick
Then I do a for each and each loop takes about a minute to execute and then assigning the values to a record takes another minute and the saving the record again a minute.

If I get a 45000 record file I am looking at days of processing whereas it should take a few minutes at most.

Any advice or tips would be great.
0
0
10 Feb 2015
Copy Link
2016-04-21 20-09-55
J.
 
MVP
It's a bit hard to tell, it doesn't sound right for sure :)

- Is it really excel, or can you use the csv component?
- And what is you for-each loop do exactly?

I am at the moment roughly doing 30000 records per minute without any bulk-inserts


0
0
11 Feb 2015
Copy Link
UserImage.jpg
Ronald Grobler
I changed the excel to record list to point at an entity rather than a structure and that improved things dramatically and I am able to continue.
But why is it that when I use a structure it takes so long to process?
0
0
11 Feb 2015
Copy Link
2016-04-21 20-09-55
J.
 
MVP
not sure.

getting an issue as well btw,  700.000 records insert from csv takes a long time..
0
0
17 Feb 2015
Copy Link
UserImage.jpg
Ronald Grobler
2 Tips I can give after struggling a lot.

1. Dont use a structure, rather import directly into a table then select from the table and process.
2. Keep the excel spreadsheets small, rather process a lot of spreadsheets rather than one big one.
0
0
18 Feb 2015
Copy Link
Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.