Trouble loading a large number of csv files

[Conlisk]

Hi All,

I am having an issue loading a large number of csv files into my LabVIEW program. I have attached a png of the simplified code for the load sequence alone.

What I want to do is load data from 5000 laser beam profiles, so 5000 csv files (68x68 elements), and then carry out some data analysis. However, the program will only ever load 2117 files, and I get no error messages. I have also tried, initially loading a single file, selecting a crop area - say 30x30 elements - and then loading the rest of the files cropped to these dimensions, but I still only get 2117 files.

Any thoughts would be much appreciated,

Kevin

[TCPlomp]

How many elements are in the array of paths (your size(s) indicator) ?

I suspect that the open file is somewhat limited to a certain number.

You could also select a certain folder and use 'List Folder' to get a list of files and load those.

Your data set is 170 MB, not really asthounising, however you should whatc your programming to prevent data-doublures.

Ton

[Conlisk]

Hi Ton,

There are 2117 elements in the size indicator.

I use the dialog to select the number of files I want to import, so I know I am selecting more than 2117 files each time I run the program. 

[Coq_Rouge]

Can you use SGL as datatype. Then you will only need half of the memory.

[Conlisk]

Just tried changing the data type there and still the same issue - only 2117 files loaded. 

I've tried an alternative way of loading the files - see attached - so as to explicitly state the number I want. However, now I get a memory-full warning when the counter reaches the constant wired to N in the for loop.

Is it possible, that when using the dialog to open a number of files, it will only load the amount it can handle within memory constraints?  

[Coq_Rouge]

You can write all the 68X68 arrays to one big binary file. And then use this file as your memory. Somewhat more programming. But at least you will not use all your memory for raw data space

[johnsold]

Is it possible that some of the files contain arrays which are larger than 68x68? If so, then a memory reallocation might occur when the larger file is read.  Perhaps do some error checking between the file read and the append to array.

Lynn

[Conlisk]

I tried that there - I can write the files to the .bin file using the hard-coding method in the last png. However, if I use the dialog only 2117 files will again be opened and stored in the .bin file. When I go to open the .bin file I then get a memory full error. 

The only option I can think of is coming up with a script in Matlab and including it as a node in LV. 

[Conlisk]

Hi Lynn, each file is exactly the same size. It's related to the number of active pixels in the CCD camera I am using to profile the laser beam. In actual fact, in my analysis program I am cropping the arrays in a pre-loading step so they are all 30x30. 

[Coq_Rouge]

What kind of analysis do you do. Do you need to have all the data in memory at all time? My plan was to have a large binary file. Then read and process data in chunkes