Read data

[pallen]

Hi there,

You might want to give OpenG's LargeFile libraries a try.

http://forums.openg.org/index.php?showtopic=114&hl=large+data

[GITA_A]

        I got this but my problem is i have a file size of 2.8GB and i am unable to read them it takes long time and it completely utilze 100% CPU usage so i want to reduce this. So pls guide me

[parthabe]

You can try out using the NI Diadem wizard which is specifically meant for large files.

[pallen]

---

AutoTEC wrote:

        I got this but my problem is i have a file size of 2.8GB and i am unable to read them it takes long time and it completely utilze 100% CPU usage so i want to reduce this. So pls guide me

---

That's a pretty big data file.  Is it all in just one big text file?

If you've already tried the OpenG tools and that isn't working for you, I'm not really sure what else I could suggest.  Personally I've never worked with data sets quite that large.

Just to open that file you're going to need a lot of RAM in your machine.  Otherwise you'll be doing a lot of hard drive swapping.  I think you could also be running up against limitations of your operating system; depending on which one you're using.

To try and limit the CPU usage, you could open the file in the SubVI and perhaps set that SubVI to a lower priority.  But it would probably take even longer to open the file that way. 

There is some information about large data sets here: http://zone.ni.com/devzone/cda/tut/p/id/3625

Good luck.

[kdmcmullan]

Ah, so the problem is not parsing the file, rather, just working with a large data set?

Hmmm... Depending on how often you have to process the file, it may be worthwhile preprocessing it. Save the data obtained by the supplied VI out to a binary file. This should reduce your subsequent read times. Maybe by about 90% since you've discarded most of the data.

...of course, if you only need to read it once, this will be a complete waste of time.

The other thing is you're trying to plot this on a graph. 2Gb of data is a lot of points. Too many, in fact. I'd guess that if all you want is a visual representation of your data, then 100 points would be too much. You might consider using the "start of read offset" input of the file read VI to skip over large quantities of your input data.

[Kodeeswaran]

I think you might have read this document which will give answers to your question.

please check the link. There was so many examples there,i feel that would be true.

https://forums.ni.com/t5/Example-Code/Managing-Large-Data-Sets-in-LabVIEW/ta-p/4100668

Managing Large Data Sets in LabVIEW

[Kodeeswaran]

if you store that file as LabVIEW Data in a folder as a binary, it might work.