Write File stops at 152 Mb

[Qualle]

Thank you GerdW for the test file.

It works very well on my machine as well, which confuses me even more.

[Ben]

Two ideas.

1) Put an indicator on the returned postion offset from your file write to see if you are over-flowing the I32 representation.

2) Move your file writes to a seperate loop and pass updates via a queue. Since you are gather data at a sizable fraction of the PCI bandwidth, the systems file cache may be getting filled. This may not show up in the stand-alone test because the DAQ is not keeping the buss busy.

Ben

[Ben]

And one more idea...

Use Gerds example to "pre-write" the file then after closing and reopening it, use that file as your data file. The theory behind this idea is that the allocation of the file is slowing down your file writes so by pre-writing it, the space for the file is allready allocated and you are just over-writing the data.

Ben

[Qualle]

You are right that the problem is related to the high sampling rate. As long as the sampling rate is below 400 kHz there are no problems. However, when going above 450 kHz the program hangs when reaching 152 Mb independently of the frequency.

/Qualle

[Kevin_Price]

Taking one giant step back to try to see the forest for the trees...

What's this app all about?  What is the meaning of the timestamps you are collecting?  What kind of processing and analysis will be done on them?  At what rate do your Sample Clock (aka "Gate") signals come in?

Counter tasks require frequent PCI bus access.  Your board and many others have an *extremely* small hardware buffer on the board for counter samples -- 2 samples worth. So the board's gotta negotiate PCI ownership almost constantly to keep that data moving into your task's RAM buffer.

With 2 counter tasks contending for the PCI bus, and the CPU demand of high-rate disk streaming, you're really up against it.  No guarantee that you can get there from here reliably (depending on exactly where here and there are...).  You may need to look into ways to re-think the app to reduce the sampling rate.

-Kevin P.

[DFGray]

The error message indicates that you are taking data faster than the system can store it to disk.  There are several things you can do about this, but it may come down to needing a RAID array or something similar.  Things to try:

1. Your chunk size writing to disk should be very close to 65,000 bytes (assuming you are running in windows).  The farther away from this you get, the slower your disk writes will be.  Getting bigger is better than getting smaller.
2. You will need to experiment with your chunk size reading from the device.  Optimums for this are generally on the order of 300kBytes.
3. You will notice that the digitizer chunk size and the write-to-disk chunk size optimums are different.  You will need to double buffer your data.  Put the read in one loop, the write in another.  Pass the data from one to the other using a queue.
   
4. Defragment your disk.
   
5. Use FAT32 instead of NTFS (minor improvement)
6. Get a faster disk or RAID array (method of last resort)

Before you do anything, check the specs on your disk drive.  If it can't keep up, you are hardware limited and will need to change something.  If you use the first four items above, you should be able to stream at hardware-limited rates.

Good luck.

[Qualle]

Thank you for your advice! I'll take this into consideration.

/Qualle

[Ben]

I have to agree with Dr Gray but I want to repeat that "pre-writting" the file should eleiminate much of the overhead associated with writting to the file. If there are still problems using the "pre-writting" technique, then the bottle kneck is the disk I/O sub-system and your choices are;

1) faster disk (raid)

2) Deffer file writes by buffering in memory until after the acq is done

3) Divide and conquer - Use two computers with idenical hardware and siycronize acq's so one machine starts at the same sample the other stops and flip-flop back and forth between the machines to collect the data. Post process to merge files from two machines.

4) Collect data on one machine and TCP/IP (or reflective memory) it to another machine for writing to file.

Ben 

[Kevin_Price]

Qualle,

   You're getting some fantastic tips from others here about extreme optimization of a difficult very-high-bandwidth task.  I'm learning some things here myself.  

   However, I'd still like to be "convinced" that your app really needs this high degree of complexity and optimization.  I've found that a pretty good percentage of the difficult counter/timer questions posed on the forums can be solved to the satisfaction of the poster by answering a simpler question.

   So, what is the physical meaning of the measurement?  What determines your sample rate?  Is it an external signal that generates the sample clock pulses?   Is it constant-rate or variable?   If variable, how fast can the rate change?   If you timestamp individual pulses every few microseconds, but the average rate doesn't change much over a millisecond, you may be working really hard to save huge quantities of high-bandwidth data that's largely redundant.

-Kevin P.

[Ben]

"80 MHz clock on each pulse that enters one of the gates."

I assume this is an Non-Destructive Test where the magintude postion of pulses relative to a gate are of interest.

Ben