Scan and read CSV files

[AntonioNI]

Hello, I have some data beeing stored un CSV files by some program, I would like to scan this files to get certain recordsets (for example recordsets un between dates of the time column of the file) or specific information such as unique values of a column.

I know I can simply read all the files and filter the datasets in LabVIEW, but is there a more efficient way to do it? Perhaps searching the file without making a copy and export only the required data?

I would also like to know wich is the fastest way to read the file out in LabVIEW, and how to read only certain columns.

[GerdW]

Hi Antonio,

I know I can simply read all the files and filter the datasets in LabVIEW, but is there a more efficient way to do it?

Beside doing the reading/filtering in LabVIEW you could use other software made for data analysis, like Diadem…

You need to define "more efficient way" to allow for proper suggestions!

Perhaps searching the file without making a copy and export only the required data?

Reading a file will not require to copy the file. Why do you think so?

I would also like to know wich is the fastest way to read the file out in LabVIEW, and how to read only certain columns.

CSV files are just text files with some delimiter chars inside. To read "certain columns" you need to read the whole file (or atleast "whole rows/lines") and index the columns.

Reading text files can be done with nearly all file functions…

General answer: The more you know about the file format the more can you optimize the file reading routines…

[Labuser16383]

It's difficult to provide answer without an example of your .csv. How large is one file, how many files, how many rows, column designations?

However there is no way to read only columns. On the other hand
you can read any amount of lines. This could be used if
the csv contains cols with different y size.

In the other cases you would want returning to importing one file at once using "read from spreadsheet file.vi", extract the desired columns, store them somewhere, forget about the rest of the file and proceed to the next file.

[crossrulz]

Sounds like you should be using an actual database instead of CSV files.  Perhaps TDMS files would also be more useful since you could set and read just properties that you care about and then get the column data that you care about.

The problem with CSV files (and text files in general) is that you do have to read the entire file in order to get a single column of data.  There is no way around that with the current setup.