How can I read a spreadsheet file up to 140MB?

[hanh]

I can not  read a 141MB spreadsheet file  for my application. I got a LabVIEW message : Not enough memory .

How can I do ?

[altenbach]

How are you reading the file? How many points are in it? How is the file structured? What is your LabVIEW version? Did you write the file with a LabVIEW program?

[smercurio_fc]

Additionally: What are your computer specs? How much RAM? What else have you got running?

Questions, questions.

[hanh]

I am reading my file with readspreadsheet file.vi LabVIEW 8.0. The file was created by LabVIEW 8.0, too (using write to spreadsheetfile.vi).

It includes 1500 rows, each row has 10,000 points. I attach my LabVIEW program.  I have tried it again and  number of rows =800 : it 's OK!!

[altenbach]

OK, you need to change your coding habits!

• First, get rid of that stacked sequence.
• In the loop in frame 4 you create way too many data copies and you spin the loop every nanosecond. Use an event structure and spin the loop only of one of the three controls has changed.
• That "times 1" on top creates a new data copy of the 2D array. Delete it!
• The three "index array" nodes can be combined into a single one with three outputs. Just resize it.
• Don't do all this bunding and escalating to complex data structures. Make a plain 2D array with three rows and feed it directly to the graph. Set x0 and dx via a property node ONCE.
• Use a state machine architecture.
• ...

[hanh]

OK. I really have little experiences doing with event structure. Your guidance is helpful. I am trying to improve my program.

[altenbach]

Here's a quick draft I made a few days ago. It's not fully tested, so you might have to adjust some things.

Can you show us a few lines of the spreadsheet file? How many significant digits do you have? Maybe SGL would be sufficent for the data?

A couple things should be changed. Currently, the program is linear, but it should be a state machine that allows reading new files and saving the current data without having to stop the program.

[hanh]

I have just tried my file (140MB) with your modified VI you sent me. it works well. Of course it takes more than 10 minutes to finish processing the file.

I used format %.3f when writing file.

I attach the VI that I used to create my file. I also attached the file (30MB) but in vain maybe due to too big file.

Thank you very much for your help.

[altenbach]

Well, your exclusively dealing with I32 integers (Actually, if your data is typical, all you need is I16! (4x less memory than your DBL). Now, if you would use a binary file, the file would be much smaller and reading it would be much, much faster!

Even if you want a spreadsheet file, you should stich with integers (use "array to spreasdheet string" with the desired type and write a plain text file and vice versa for reading).

The same coding rules I showed in my other attachment apply here too. You use way too many locals and sequences. None are needed!

Also many things can be dramatically simplified. For example look at your inner while loop. The entire thing can be replaced by essentially an atomic operation!

 See if it makes sense! 🙂

(for my alternative, I am using I16. Modify as needed).

Message Edited by altenbach on 04-17-2008 07:35 PM