frequency of writing data to disk

[Kousy]

Hi all,

I have a query.

At what frequency can I write data to a disk? Is there any limitation on this?

In Data acquisition application, I am using write file function to write data to hard disk at the frequency of 0.5 sec.  LV can handle this? Because my application got slowed down, over a period of some hours.

Thanks for your help in advance.

Kousy

[Elijah_K]

Hey Kousy,

How are you writing the data to a file?  There are a number of different ways and formats in which to do this in LabVIEW.  Some are faster than others and offer more flexibility and post-processing options.  TDMS is a format for binary measurement files that is faster, and actually smaller than lvm text-based measurement files.  If TDMS sounds like something that might help you in your application, find out more about it here: http://zone.ni.com/reference/en-XX/help/371361B-01/lvhowto/ni_test_data_exchange/

Otherwise, let me know what VIs you're using to write data to a file and we'll troubleshoot the difficulties you're having.

[Kousy]

Hi Elijah,

I am using write characters to file.vi & write file function. Also I have attached the vi. Hope this will help you to give a better suggestion.

Kousy

[DFGray]

Do you really need a text file or would a binary also work for you?  Text files are slow for two reasons:

1. You have to convert your binary data to text format
2. Text format is typically 5 - 10 times bigger than binary, meaning you have to write more to disk
   

If you want to use binary, check out NI-HWS or TDM (make sure you use the binary form, not the text form).  NI-HWS would be best for waveforms, TDM for anything else.  TDMS, a streaming form of TDM, is not available for LV7.1.

If you are streaming data to disk, you should not use the Write Characters to File VI.  Every time you call it, it opens the file, writes your data, and closes the file.  Opening and closing the file is a huge overhead.  This overhead is probably what is causing your long-term slowdown.  Typically, you open the file, write the data multiple times, then close the file.  The VIs you need are in the second frame of your sequence, just put the write VI in a loop.

If you are interested in speed, on Windows platforms, you get the best speed when the data size you write to disk is about 65,000 bytes.

If you have any more questions, please ask.

[Elijah_K]

Kousy,

DFGray is right.  The problems you're having stem from writing to the file every time you get data.  Get all your data first and then send it to a file.  If you think about what's going on, you're asking your computer to open, write and close a file over and over again, which is not a good practice.  The delays you're experiencing happen because any windows computer is not deterministic and delays can be introduced in software timing due to any other process that demands processor time.

I hope this makes sense.  It sounds like you might want to consider a streaming file format like TDMS

[Kousy]

Yes DFgray, definitely I need to save data in text format. Endurance application badly need this, once the day starts, assembled the testing unit & starts running the rig, it goes on running till the day ends or any emergency happens or failure occurs. so this is the ONLY data available to analyze why the failure happens.

I used Write waveforms to file.vi also for about one hour gap or so to capture all critical parameters as waveform.

I am not aware of TDMS.

"Typically, you open the file, write the data multiple times, then close the file.  The VIs you need are in the second frame of your sequence, just put the write VI in a loop." - I cant understand. want to accumulate the data in a array & write it at the end or what? pls help.

But Elijah, this data capture does not take much cpu resources i think, becos say inclusion of data capture in a vi takes 45% of cpu usage means, commenting out this takes around 35-36% of cpu usage. i hope this hardly matters.

Kousy

[Elijah_K]

Kousy,

Your system's processor consumption is not a direct indication of the speed at which the application will execute.  Processes and data can only be moved through your system's bus at a certain speed depending upon its availability and the speed of the other devices involved.  When writing data to a disk the bottleneck is most definitely your hard drive.  It is not a good idea to continuously write data to a disk.

Data that is saved to disk in a binary format such as TDMS can later be changed to a text format.  There are countless post-processing options for data acquired using TDMS.  A streaming binary format is ideal for your purposes if your application is running for an extended period of time and isn't supposed to stop or slow down during the course of the day.  Writing to a text file is going to be next to impossible to do continuously over the course of a day.

For more information about TDMS and how to use it, check out this DevZone article:
 http://zone.ni.com/reference/en-XX/help/371361B-01/lvhowto/ni_test_data_exchange/

Also, several examples ship with LabVIEW that demonstrate how to use it.  These include examples of how to open a TDMS file and retrieve the data - at which point you would easily be able to convert it all to a text file.  Just open the Example Finder in the help menu and search for TDMS.

Let us know if you have any further inquiries.  Thank you!

Message Edited by Elijah K on 12-10-2006 11:31 PM

[DFGray]

Elijah, Kousy cannot use TDMS because he is using LabVIEW 7.1.

Kousy, I have attached a VI that shows the basic idea of what I was trying to explain above.  It uses a producer/consumer architecture.  Note that the producer and consumer loops can be in different VIs.  Just pass the queue reference around to make it work.  This VI will store data until you run out our disk space, provided you have an NTFS file system.  It should not have any slowdown issues over long time periods.  If you want to speed it up more, put some logic in the consumer loop so you only save when you have 65000 bytes to save.  However, at your data rates, this should not be necessary.

Let us know if you need more help.

[Kousy]

Thanks DFGray.

I have NTFS system only. I can make use of the vi you sent I think.  But I have not used this queue reference functionality till date. So I'll try to understand and can proceed hopefully.

I told earlier I used Write waveforms to file.vi in my application. The attached vi is the one I am using as subvi.

I'' be calling this subvi in a for loop (to capture 8 parameters from Daqmx AI read array, jpg attached) once in 50ms period, for total period of 5 seconds or so. Calling subvi many times in a vi will slow down the application? Need help.

Kousy

[DFGray]

From a quick look, it appears you can use autoindexing on the array input to the FOR loop to do what you are doing in the case selector.  I cannot read the VI you posted because it is passwork protected.  Please repost or post the password and I will take a look.

Also, check out the LabVIEW examples for how queues work.  They are essentially a FIFO buffer which can be read anywhere you can pass the queue reference.  Care is necessary, because if you read data at one location, it will not be available at another.  However, they are perfect for producer/consumer loops.