How to read blocks of 100 or more rows from a text file using read from spreadsheet file.vi?

[wasabi22]

I have a large file containing 9 columns and a lot of rows. The file size is about 170 MB.

When I read the file by using read from spreadsheet file.vi, I get the Labview Memory full error.

To avoid this error I'm thinking to read the first 1000 rows and then another 1000 rows( with an offset of 1000) 

and so until the file ends. So i would like to continuosly streaming entire file to analog outputs without interruption.

How do you do this?can i avoid this error by using this method?other methods?

The txt and vi its attached. 

Thanks for any help! 

Hi,

I have a large file containing 9 columns and a lot of rows. The file size is about 170 MB.

When I read the file by using read from spreadsheet file.vi, I get the Labview Memory full error.

To avoid this error I'm thinking to read the first 100 rows and then another 100 rows( with an offset of 100) and so until the file ends.

How do you do this?can i avoid this error by using this method?other methods?

Of course i need to continuosly streaming part of entire file to analog outputs without interruption.

I attach only a part of the text file because it is too big.

The zip its attached. 

Thanks for any help! 

[aeastet]

I took your txt file and put it into a read spread sheet function and it work fine. The problem is somewhere else in your code. Did you use the light bulb to see where you are having the problem?

See my attached code that only opens your data file.

[aeastet]

When I run your code on my computer I have no problems. The data file is not that big. Even MS Excel can open it without issue.

[smercurio_fc]

As the poster noted, the text file that was included in the zip file was only a small sample of the actual file, which is about 170MB. This could be a problem trying to read it into memory since there will be several memory allocations going on, and I can see an out of memory occurring.

To poster: It seems that you abandoned the pre-allocation method I showed you in your original thread. That is another reason why you're getting an out of memory error. You are using Build Array inside the loop with HUGE arrays. I would strongly suggest reading this article: Managing Large Data Sets in LabVIEW.

As for reading the file in chunks, you can do this using the Read Text File function rather than the Read From Spreadsheet File VI. If you right-click on the function you can select the Read Lines option. You can place that in a loop to read a certain number of lines and process the data.

[wasabi22]

Thanks for the information Smercurio. I have read Managing Large Data Sets in LabVIEW but i don't find a solution. 

I haven't abandoned the pre-allocation method. It was a old version. My current program is attached. But I always 

have the same Memory full error using file of about 170 MB.

The error occurs even if I only read file (170MB) with read from spreadsheet and graph, without built array and

without Preallocation method.

To avoid this error I was thinking to read the first 100 rows and then another 100 rows and so until the file ends.

I don't understand yet how to read  some lines into a loop using read from text file changed in lines:-(

So if I read a entire file in chunks  

can i make a generate data continuosly, without interruption or delay buffer between last sample of a part of file

and first sample of next part of file?

I hope I'm clear enough...

Any help would be appreciated.

Thanks for the information Smercurio.

I have read Managing Large Data Sets in LabVIEW but i don't find a solution. I haven't abandoned the pre-allocation method. It was a old version.

My current program is attached. But I always have the same Memory full error using file of about 170 MB.The error occurs even if I only read file (170MB) with read from spreadsheet and graph, without built array andwithout Preallocation method.To avoid this error I was thinking to read the first 100 rows and then another 100 rows and so until the file ends.I don't understand yet how to read  some lines into a loop using read from text file changed in lines 😞

So if I read a entire file in chunks  can i make a generate data continuosly, without interruption or delay buffer between last sample of a part of file and first sample of next part of file?

I hope I'm clear enough...

Any help would be appreciated.