What can I do to write a binary file to acquire all acquired data?

[coreyGN]

I'm having a problem writing to a binary file. I'm writing 24 arrays, about 35,000 values to a I16 binary file. I've read "Problems with continuos sampling and writing to disk PXI-6071E" in this forum, but that was not solved. I'm reading from 24 arrays from 24 RT FIFOs and putting those into a 2D array. Then I write that 2D array using the Write File [I16] vi. The problem is that it's writing only once every 3 or 4 seconds. The RT FIFO gets overwritten in no time from the acqusition vi.

I have a Communication vi (created by the RT Communication Wizard and modified) and within that vi i have the Writing vi and an acquisition vi. Here are my specs:

Dell GPX Optiplex 270
Around 3.0 Ghz
Around 512MB RAM
Reading from 3 PCI 6143 boards, 8 channels each
Sampling 250K samples/sec
Reading 35,000 samples per read
Hard-drive space about 70GB

Here's what I know:
- Replace Array is more efficient than Build Array, so i'm using Replace Array (initializing a 2d array outside the loop)
- writing in multiples of 512KB is more efficient... I think i tried this, but no difference.
- The hard drive should be able to write at this speed.
- The acquisition vi might be using all the processing time... if so, how do I resolve my problem?

[aartjan]

Not sure whether my answer is what you are looking for, but in general terms, this is what I think you should keep in mind:
there are three "memory-stages" in acquiring data:
-samples are taken by hardware and put into small on-board FIFO memory,
-Labview grabs samples from FIFO memory- the samples are now in the computer's RAM,
-Labview writes samples to the hard drive, allowing RAM space to be overwritten by new samples.

Writing to the hard drive is a rather slow process, so you want to write big blocks of samples at at once.
What you are saying about the replace array function is very imortant: make sure that Labview allocates all the RAM memory it needs for the samples ahead of time. You do that by initializing an array of sufficient size. When it comes close to being full, you make your program write the whole array to the hard drive at once.
Make sure you create a large enough FIFO buffer on your DAQ card to survive milliseconds of delay due to software timing and sudden CPU loads. Next thing that you can do is to first find out how many samples are present in the FIFO and then immediately grabbing them all. This prevents your daq vi's from waiting until all requested samples are present on the FIFO.
Another thing that I do: I look how much extra time I have left over after Labview has performed all its tasks. If there is plenty of time left, I can have Labview write samples to disk. If not, I make it go grab samples from the FIFO again.

As for the binary problem: Use the write file function and feed it the 2d array directly: that will give you a binary file without messing with the datatype etc.

[nickf]

You should also consider the actual data rate at which you are attempting to write data to the hard disk.
Based on the specifications that you posted I calculate that you are approaching 46MB/sec of data streaming to disk if writing double (DBL) values to disk. IDE hard drives can typically sustain a datarate of anywhere from 20-40 MB/s. SCSI harddrives can usually go a litte faster, 30-58 MB/s.

You also have to consider what else the computer is doing while it's writing to disk. You mentioned that the data acqusition loop is taking a lot of processing time. --Have you added a wait VI in your data acqusition loop? Also, have you considered using fewer RT FIFOs? Reducing the number of FIFOs will reduce the overhead associated with maintaining all of that data in memory.

The type of data you're writing to disk will also effect how fast you can stream that data to disk. The LabVIEW DBL data-type is 64-bits. A single (SGL) is only 32-bits. This would cut your data transfer rate by half, down to 23 MB/s. Since your data acqusition board is only 16-bits you should be fine using SGL values. If you decide to use this method you should convert you data to SGL format immediately after you get it back from the DAQ read function and before you send it through the RT FIFOs. If you're really crunched for data-rate you could even read back unscaled values from the board. This would return 16-bit data and further reduce your datarate to about 12 MB/s.

Note: If you read back unscaled values in DAQmx you will also need to save the device scaling coefficients. You can do this with a Channel Property node. In the Channel Property node choose: Analog Input:General Properties:Advanced:Device Scaling Coefficients:Device Scaling Coefficients. You use the scaling coefficients to calculate voltage from your unscaled data.

[coreyGN]

Thank you very much for the descriptive solutions. I haven't been able to try them yet, but I will soon. I'm planning on reducing my 24 fifos to one large array (about 30,000 x 24 16-bit values). Tychova, it would be very helpful if I could get 16-bit values off of my 6143 board. I found the coefficient property node, but how do I get the board to give me 16-bit values instead of DBL values?

On a side note, I created and ran this vi on both the Host machine and the Target machine. The vi runs 4 times slower on the target. Is this normal? Is this relevant?

Thanks.

[nickf]

You can read unscaled values from your DAQ board by choosing the appropriate instance of the DAQmx Read VI. You will probably use "Analog->Unscaled->2D I16" or "Analog->Unscaled->2D U16". I would use I16 for data that was both positive and negative, and U16 for only positive or only negative data.

With regard to the VI you created you should keep in mind that Windows is a multi-threaded environment and LabVIEW RT is a multi-tasking environment. You should generally expect better performance for VIs that are run in Windows. Running your VI on a real-time system will give you the assurance that your application will run at a relatively constant rate for extendend periods of time, and will generally not be interrupted by other processes running on the same machine. The VI you attached is also performing a relatively large amount of data manipulation in the same loop with the disk IO which is largly system dependent. Of course, you also have to consider the general specifications of the RT target machine.

One other very critial step in ensuring reliable performance on your RT target is to include a "Wait Until Next ms Multiple" VI in any loop that runs in your RT application. Since LabVIEW RT is multi-tasking different tasks have no way to take control of the processor if they need to execute something. Adding a different wait-time for each loop gives you the ability to prioritize all the loops in your application. Attempting to run even one loop without a wait function will give you unpredictable results because there are several background tasks running on the RT target machine. Examples of these would be TCP/IP communication and LabVIEW memory management.

[ODRulz]

Hi nickf,

I have a question involving using the RTFIFO's with the Unscaled->2D I16 data.  How? 

I have a PXI-6052E that I'm trying to get binary data off and place into a RTFIFO in my read loop, then read the RTFIFO and put that into an array that is written to a file only when there's 512Bytes of data in another loop.  The "RTFIFORead I16a.vi" to the DAQmx Read Unscaled2D I16 vi don't connect properly because of the array size differences?  How, or what RTFIFO's do I use for the unscaled 2D I16 data?

Also, do you know where I can find out whether the 6052E puts it data out interleaved or non-interleaved?  Help says "in the documentation for the device", but I didn't get any documentation and I couldn't find that information on the datasheet.

Thanks for any help.

[Ame_G]

ODRulz,

It is not possible to write 2D arrays to RT FIFOs. In your case, you were reading 2D arrays using DADmx Read VI and were trying to write it to a FIFO. The work around is to use Index Array VI and index each row or column and right the indexed 1D array to a FIFO. I have attached a VI which will show you how to wire from the DAQmx Read VI to the RT-FIFO. Please read this link for more information about RT FIFOs.
http://zone.ni.com/devzone/conceptd.nsf/webmain/3a9b9598f6c4d06186256ab8004f6340
As to the data you are reading from the DAQ board they are interleaved if you are reading from two or more channels.

Regards,
Ame G.
National Instruments

[ODRulz]

Hi Ame,

Thank you for your help first of all.  I was pretty sure that a 2D array couldn't be used with the RTFIFO directly, but not 100%.  Thanks for clearing that up.

As far as using the 2D array, what would each row of data be if I split them up as you showed in your attachment? (ie, would row 0 be an array of channel numbers over and over & row 1 be all the data, or something else entirely?)  My understanding of using the 2D array were that all of the data for each channel (2 or more channels) would be output in it's own row.  I guess I'm not exactly sure how the data is coming out of the read vi for the 2D array.

Thanks again,

Scott

[Ame_G]

Scott,
As you have mentioned when you are reading from multiple channels each row will have the value a channel. For example, if you are reading from two channels, channel 0 and chanel1, the first row represents the reading from Channel 0 and the second row from channel 1.

Regards,
Ame G.
National Instruments