Importing large quantities of data

Importing large quantities of data


Please can someone give me some advice on how to process files faster.

I am having problems importing data.
The current file I am having problems with is an excel document has 190 columns of data, I removed all but 10 rows of data.
First, I convert the data into a record list. This is quick
Then I do a for each and each loop takes about a minute to execute and then assigning the values to a record takes another minute and the saving the record again a minute.

If I get a 45000 record file I am looking at days of processing whereas it should take a few minutes at most.

Any advice or tips would be great.
It's a bit hard to tell, it doesn't sound right for sure :)

- Is it really excel, or can you use the csv component?
- And what is you for-each loop do exactly?

I am at the moment roughly doing 30000 records per minute without any bulk-inserts

I changed the excel to record list to point at an entity rather than a structure and that improved things dramatically and I am able to continue.
But why is it that when I use a structure it takes so long to process?
not sure.

getting an issue as well btw,  700.000 records insert from csv takes a long time..
2 Tips I can give after struggling a lot.

1. Dont use a structure, rather import directly into a table then select from the table and process.
2. Keep the excel spreadsheets small, rather process a lot of spreadsheets rather than one big one.