Csv file problem

[Citras]

Hi,

I am working on data acquistion and I have problem with the csv file size,

since the data acquisition is continuos and I run it for days, It consist at least tens million of row. But I can´t save it into 1 excell file since excell file maximum row allowed is 1048576rows. Is it possible to save the data into csv file and when it reached the maximum rows then it save to other file?( it has to be automatically since the data continuosly runs for days )

Or maybe there´s better way to solve this case?

Best regards,

Citras

[GerdW]

Hi citras,

1st better way: Don't use Excel when it comes to big data management.

2nd better way: Don't use Excel... 🙂

- Yes, it's possible to create new files when the current grows too big. You have to program that...

- Use tdm(s) for huge data files. There's a (free) plugin for Excel to open such files from Excel.

- Use Diadem (from NI) for working with huge data files.

- Use Origin for working with huge data files.

Need more hints?

[Wayne.C]

Citras,

You could just keep track of how much data has been written to a file and start a new on when the desired file size/# lines is reached.  Any code I write that is going to log data for extended periods of time is designed so that a new log file is automatically started at least once a day and sometimes more often depending on the amount of data logged.   I do this to control file size and minimize the potential for data loss if a file were to become corrupt. I tend to use time to determine when to start a new file. 

As Gerd has indicated, there are much better choices than .csv as well.

[Citras]

Hi,

So, which data files is the best if I need to manage huge data files? Can you explain how to save data into those file format??

Thank you,

Best regards,

Citras

[smercurio_fc]

Did you read Gerd's reply? He gave you three possibilities.

For TDMS you have VIs in LabVIEW to read/write TDMS files. Examples ship with LabVIEW. Open them. Look at them.

For Diadem: https://www.ni.com/en/shop/data-acquisition-and-control/application-software-for-data-acquisition-an...

For Origin: http://lmgtfy.com/?q=origin+data+file

[GerdW]

Hi citras,

also asking for the "best" solution without providing any constraints is quite senseless...

[Citras]

Hi,

I have read all your suggestion and I am interested in using Diadem,

I have look at the link smercurio_fc suggest and it looks like Diadem is a sofware to simulate the signal base on excell data.

Can you explain how Diadem can solve my problem since it simulate base on excell data ( which is have maximum of 1048576 rows )?

Also I attach my block diagram for data acquisition

Best regards,

Citra

[SimonH]

I highly recommend exploring DIAdem if you regularly work with large data sets.

Having said that there's an easy way to get the behavior you are looking for.  Instead of using the Write to Spreadsheet file VI, use the Write to Measurement File Express VI.  In the advanced options of the express VI you have options for splitting into a series of files after various conditions are met.

An .lvm file is an ASCII file you can open with Excel (or Notepad etc.) that has additional header information.  TDMS is a more efficient file format that can be loaded into Excel using a free plugin that installs along with LabVIEW.

Regards,

Simon

[Suse]

Hi Citras,

the best way to save large data sets is to use a TDM or TDMS file. If you create new files after a specific number of rows you can also open them in Excel with the TDM Excel Add-In.

TDM Excel Add-In for Microsoft Excel Download: https://www.ni.com/en/support/downloads/tools-network/download.tdm-excel-add-in-for-microsoft-excel....

In LabVIEW there are specific VIs for TDMS reading/writing.

If you use TDM oder TDMS you can also open the files in DIAdem. This is a software for loading, analyzing and reporting data.

I hope I could help you.

Best regards

Suse