"Data was overwritten before it could be read by the system" on PCI-6259

[Dave Rahmlow]

Hi-
   I have two VIs which work fine independently, but together cause one of the programs to stop (after a short time) with a -200141: "Data was overwritten before it could be read by the system." error message.

One of the two programs sets up analog output voltage waveforms (finite samples, using onboard memory) on 3 analog output channels of the PCI-6259 board.  It then sets up an analog read (synchronized with the analog output clock), and starts the analog write/read tasks.  It repeatedly does this write-read cycle until the program is stopped.  This program does not generate any errors.  I tried changing the AI.DataXferReqCond for the analog read, but it didn't fix the problem.

The second program counts two sets of digital pulses using continuous buffered counting mode.  A separate counter (on a PCI-6723 connected via RTSI) generates a pulse train of 500 pulses at 100 kHz once a start trigger is received; start triggers are received at 20 Hz.  The output of this counter is used as the clock for the buffered counting mode on the two counters on the PCI-6259 board.  The signals being counted on these two counter channels are about 20-200 kHz.  This program runs fine on its own, but generates a "-200141: Data was overwritten before it could be read by the system" error in the "daqMX Read Counter 1D N sample" vi after working for a short time (about 2 to 10 seconds) if (and only if) the other program is also running.  With both programs running, the Windows task manager reports cpu usage at below 8%.  The Channel Property node reports that the counters are using DMA.  Changing the input buffer size does not help (I do monitor how full the input buffer is; this error can occur when the input buffer is nearly empty).

I am using Labview 7 with daqMX 8.1.0f1 on XP.
Any suggestions?
Thanks-
   --- Dave

[Chris_D]

Hey Dave,

They error that you are receiving indicates that the counter buffer is overflowing. The specs for the 6259 indicate that the counter FIFO can only hold 2 samples. This means that as soon as the counter data is put into the FIFO, we need to immediately transfer it to the PC buffer. It looks like when your application runs the counter VI on its own then it can meet these requirements, but when you additionally perform Analog Output operations then there is not enough PCI bandwidth to transfer the data from the counter FIFO fast enough and the overflow occurs.

You may be able to avoid the error by decreasing your sampling rates. Really anything that will decrease the use of your PCI bus time will help the performance. Do you have any other PCI cards in the computer that you could remove?

NI 625x Specifications
http://digital.ni.com/manuals.nsf/websearch/210C73CBF91128B9862572FF0076BE85

Please let me know if you have additional questions.


Message Edited by Chris_D on 06-03-2008 02:19 PM

[Dave Rahmlow]

Thanks for the reply-

yeah, I had seen that the counter FIFO is only 2 deep, and references to limited PCI bandwidth.  But, the analog task should not require that much additional bandwidth (2 kHz sample rate * 6 channels * 4 bytes/sample = 48 kBytes/second, compared to the "ideal" PCI bandwidth of 133 MB/sec, is ~ 0.03% of the available bandwidth).  (If you include the analog output part of the task, 2 kHz * 3 channels * 4 bytes/sample is another 24 kB/sec - though it should not be rewriting this data each time, since AO.UseOnlyOnBrdMem is true.)  It doesn't make much sense to me that the counter will *always* work fine if the other program is not running, but fail if it is.  (Is there a windows tool that will show PCI utilization?)

There are a couple other cards in the computer (multichannel scaler, NI GPIB card) but these are very low bandwidth (< 100 bytes/sec) and are used by the counter program without interference.  The video card is an AGP Radeon 9800.

Could there be some issue of priority, eg. the counter is assigned a lower priority DMA channel than the analog task, or perhaps is quietly assigned to an interrupt-driven routine instead?

I did try running the analog write/read vi with just the read and with both the write and read disabled (but the rest of the logic running).  Eliminating the read allowed the other vi to run about 2 minutes before failing; eliminating both parts allowed the vi to run properly.

Thanks-
  --- Dave

[Dave Rahmlow]

Sorry, quick correction to what is above...the cpu usage of the counter task is about 70% on its own; the utilization of the analog task is about 7%.  However, heavy system load (100%) during the counter task alone doesn't cause it to fail.

I tried switching one or both of the analog read/write tasks to Interrupts from DMA.  Switching either or both allows the counter task to run for about 40-100 seconds before failing, similar to disabling analog reads altogether.  But switching both to interrupts doesn't allow the counter task to run indefinitely.

[Chris_D]

Hey Dave,

I don't know of a Windows utility that will show PCI utilization. Also, I don't believe there is a priority issue with the counter DMA channel.

If you wouldn't mind, I would like for you to try a couple more tests. First, let's try increasing the sample rate of buffered counter inputs without the AI/AO VI running. See if you can increase the rate to a point where the error occurs. This will show that there is ultimately a PCI bandwidth issue even without the AI/AO VI. The AI/AO program must be using just enough bandwidth to cause the error with your original settings.

Next, we can try decreasing the sample rate of buffered counter inputs, but do this with the AI/AO VI running. See if you can decrease the rate to a point where the error does not occur. Hopefully this will show you a point where you can run the program without error.

Message Edited by Chris_D on 06-06-2008 01:20 PM

[Dave Rahmlow]

Hi-
when I run the AO/AI vi, the buffered count rate (bin size) needs to be increased from 10 to 35 uS in order to reliably prevent an error.
when I don't run the AO/AI vi, the buffered count rate can be brought down to at least 1 uS.

I also tried running the counter program, then running the analog output task alone from within Measurement Explorer.  About 1 time in 5, running the analog output (finite samples mode, internal clock, 1000 samples at a 1.25 kHz rate on 3 channels) kills the counter program if the bin size is 10 uS.  (Doing analog output alone seems to be enough to cause problems).

Another variant on this - I commented out the AI task in the AO/AI program.  So, it sets AO.UseOnlyBrdMem to True, writes a waveform to the card, then repeatedly calls DaqMX Start/Wait Until Done/Stop Task on the AO task without changing the waveform.  (This shouldn't involve any significant data transfer on the PCI bus after the initial write to on-board memory).  If I then start the counter program, it still generates the error after a short time.  If I comment out both the AI and AO Start Task vis, the counter program runs fine.
   Thanks-
  --- Dave

[Chris_D]

Hey Dave,

It looks like I was mistaken and the limitation is not with the PCI bandwidth, but a hardware limitation with STC timing chip. Knowing this, there still seems to be something fishy going on with your application. You can see from the benchmarks link below that your device should be able to perform at about 380 kHz. I was able to reproduce similar results with my 625x device.

Benchmarks for Buffered Counter Input with M-Series Devices
http://digital.ni.com/public.nsf/allkb/72A7E41EE5A8756A862571DA0076F1D7?OpenDocument

When you say 1 uS I am assuming you are meaning a sample rate for your buffered counter input is 1 MHz. Are you sure this is correct? The benchmarks and my tests indicate that you shouldn't be able to achieve near this rate.

It is still very strange that you get the error just doing a simple AO with MAX. Have you tried testing with any of the example programs? This would ensure that we are using the same software in our tests. I am using the Count Digital Events-Buffered-Continuous-Ext Clk.vi. Since we have now determined that this an issue with the STC chip this would explain why it still generates the error even when you write the waveform to memory before starting the counter VI.

We also may want to try testing with a continuous 100 kHz sample clock source from your counter rather than one that creates 500 pulse every 20 Hz. In my testing, I saw the error when changing the rate of my sample clock and am wondering if you may be seeing something similar.

To really solve this issue we need to step back and take a look at the big picture. What is your requirement for this application? Will a slower rate be sufficient or are you just curious to cause of this error? Could you use polling instead of buffering? Would you consider a different device with larger FIFO?

[Dave Rahmlow]

Hi-
so, I tried using a few of the example programs (Gen Mult Volt Updates-Int Clk.vi  to do some analog I/O, Count Digital Events-Buffered-Continuous-Ext Clk.vi to do the counting, and Gen Dig Pulse Train-Continuous.vi to do the 100 kHz clock).  This uses a continuous 100 kHz clock (I tried the 100kHz timebase, and a counter on a different card).  I made a couple minimal modifications to the examples (put a while loop around the innards of the analog ouput vi to keep it running, added a control to set the output terminal for the pulse train vi).  The screenshot is below:



I found that the system works fine if I keep the analog output Desired Frequency/Samples per Buffer at or below 10 Hz (eg. 2 updates/sec)/1200 samples.  Going much above that causes errors...so, I can get errors without doing anything too demanding in terms of analog output.

Can you replicate something like that with the

If I use this setup, I find that I can set the frequency up to about 1 MHz sample rate for the buffered counter read, but it can only sustain this rate for a limited time (few hundred thousand samples).  I think the difference is that in my program, I only generated blocks of 500 samples at a time so the total throughput was fairly low.

Overall, I can work around the issue if I have to.  The 2 counter tasks are being used to read in photon arrival times from two photomultiplier tubes, and are the most important part of the experiment.  The analog tasks are being used to monitor and control some laser hardware; I could use an old-fashioned benchtop function generator and oscilloscope instead of the DAQ card if I have to.  But, if there's some clear reason why I'm getting a problem here, it would be nice to fix it the right way.

   Thanks-
   --- Dave

[Chris_D]

Hey Dave,

This is good now that we have the same test setup. I was able to test with your exact settings and was unable to produce the error. This at leasts proves that something is going wrong here. I will check with R&D and see if they can provide any insight.


Message Edited by Chris_D on 06-12-2008 05:58 PM

[Chris_D]

Dave,

I was able to get a few more suggestions from R&D. We don't spec buffered counter input because it can vary depending on the system. One way for us to test if this is a PCI bandwidth problem or internal hardware limitation would be to test the AO with regeneration. This way we transfer the AO waveform data before we start the CI operation, thus guaranteeing the AO is not using the PCI bus. I believe you already tested this before, but we may want to try again with our current setup. If you do still receive the error with regeneration then this does indicate that it is a limitation of the hardware or could be a problem with your particular device as I am able to perform the tests without error.