Maximize rate of continuous digital data logging

[Trehcir]

I am trying to log large amounts digital externally clocked data quickly.  The data is solely for post processing.  What is the best approach to get data from the port to the hard drive at the fastest rate possible?

I am logging using a PCIe 6537B.  I am logging on 4, 8-bit ports at 40.92 Mhz, so a rate of ~160MBytes per second.  I have two designs that I have tried thus far (a producer-consumer architecture and a DAQ.mx configure logging design).  Both designs work if I only log on two ports (effectively halfing the throughput needed) but fail at the 32-bit width.

My producer consumer architecture fails after around 30 seconds of successful logging with  the message

Error - 200613: Acquisition has been stopped to prevent an input buffer overwrite.

My alternative DAQ.mx configure logging design fails very quickly (~1s) with

Error 200361: Onboard device memory overflow

.

I might be configuring it incorrectly.  Like I said, It works with a data width of 16 bits.  I don't know if the increase of data width is directly the problem or if the increased throughput is causing the failure.

I am running this on a 3GB ram window's xp using labview 2012.  I attached the vi's of both my attempts and an image of the DAQ.mx configure logging design.

[Hooovahh · Accepted solution]

Well I'd suggest the built in TDMS logging method would probably give you the best results, assuming it works at all.  That type of logging opens a DMA channel from the hardware to the diskdrive so it shouldn't produce unnecessary data copies, which a producer consumer loop probably would.

I've never tried logging to TDMS without reading the data, does a DAQmx Read need to be used in the loop?  Does your TDMS file contain any data?

Is your harddisk up to the challenge?  Is it a laptop platter drive?  I'd recommend testing on an SSD if possible just to see.

There may also be properties to open up a buffer size.

Start with one of the basic TDMS logging examples that ship with LabVIEW and work your way up from there.

[Trehcir]

Yes, the written file contains about 160MB, which is about how much I expect after 1 second.  I can log 16 bits per sample for minutes, logging GB of data (but I need all 32 bits).  In fact, I would have to change the setting to "read and log" for labview to allow me to use "DAQ read".  Labview's documentation claims I can write faster if I tell it I won't read the data and am just post processing.

Are there any tests or checks I could do to determine if the hard drive is the problem?  I will ask around the lab and see if I can borrow a SSD.

[Hooovahh]

---

@Trehcir wrote:

Labview's documentation claims I can write faster if I tell it I won't read the data and am just post processing.

---

I believe that, I just never tried writing to a file, without reading it, I wasn't sure if it would only log if it was told to read.  Apparently not.  The fact that a file is made is a good sign.

Now that I look at modern harddrive speeds I'm not sure that is your issue.  Shipped with LabVIEW is a few TDMS write testers.  Help >> Find Examples search TDMS Speed.  My laptop drive has 427.6M/second and while it isn't old it isn't an SSD either.

[Trehcir]

I ran the speed test as you recommended.

I get 180MB/s (I defragged the disk before the test).

I need about 160MB/s, so making this work should be theoretically possible if I'm not adding any overhead, but the margin is small enough that I wouldn't be suprised if this was the problem.  The PCIe bus I'm using is only suppose to go up to 200MB/s as well, so I'm cutting it close at several steps in the pipeline.