How do i open a VERY big file?

[therealkilkenny]

I hope someone can help.

I did some testing using a LeCroy LT342 in segment mode. Using the Labview driver i downloaded the data over GPIB and saved it to a spreadsheet file. Unfortunately it created very big files (ranging from 200MB to 600MB). I now need to process them but Labview doesn't like them. I would be very happy to split the files into an individual file for each row (i can do this quite easily) but labview just sits there when i try to open the file.

I don't know enough about computers and memory (my spec is 1.8GHz Pentium 4, 384MB RAM) to figure out whether if i just leave it for long enough it will do the job or not.

Has anyone any experience or help they could offer?

Thanks,

Phil

[Sheldon_Stokes]

Rather than use the high level spreadsheet read VI, you probably need to read your data in a line at a time using the lower level open- read (til EOF) - close.  For such big data sets you should really consider saving the data in a binary format, you will suffer less.

Sheldon

[falkpl]

When you open (and read) a file you usually move it from your hard disk (permanent storage) to ram.  This allows you to manipulate it in high speeds using fast RAM memory, if you don't have enough memory (RAM) to read the whole file,  you will be forced to use virtual memory (uses swap space on the HD as "virtual" RAM) which is very slow.  Since you only have 384 MB of RAM and want to process Huge files (200MB-600MB) you could easily and inexpensively upgrade to 1GB of RAM and see large speed increases.  A better option is to lode the file in chunks looking at some number of lines at a time and processing this amount of data and repeat until the file is complete, this will be more programming but will allow you to use much lass RAM at any instance.

Paul

[therealkilkenny]

Sheldon and Paul,

You guys are great. I found that if i read in the first line of the file and find the end of line character i can then use that as the marker to read in the next line and find the next EOL character. Thus i progress through my file and get it into useful data.

Thanks for your help.

[Tomi_Maila]

The file size was not a limiting issue with the asker of the question. However if the file size exceed I think 2 GB, normal routines doesn't work anymore. There is a openG routine for handling very large files. You can download openG from http://openg.org.

Tomi

Message Edited by Tomi M on 11-21-2005 05:55 PM

[Ray.R]

Hi Tomi,

I seem to recall the same value... 2GB limit. 

A "fairly" large file, nevertheless..  😉

[DFGray]

The 2GByte limit does not exist for LV8, provided your file system can handle the larger files (e.g. FAT32 can't, NTFS can). For more tips and tricks on handling large files, check out Managing Large Data Sets in LabVIEW.