Delays in DAQmxWriteRaw when using multiple PXI-6733 cards

[rmattes]

Application:
I have an application that generates and writes analog data at slightly below 300Khz.  The data generation and output takes place every 14 ms (we'll call that one "block",) via an EveryNSamples event from a timing card.  At every block, 4000 samples of analog data are generated and output to 6 of the 8 analog output channels of the PXI-6733.  The output buffer is set to twice the 4000 sample write block, and pre-filled with 4000 zeros, so the output is always one block behind the generated data.  If any single block takes longer than 14ms to generate and write out the analog samples, the output buffer will underrun and output stops (regeneration is turned off.)  I recently needed to add 10 more channels of output, so I installed a second PXI-6733 card into the chassis, and set up a second analog output task to write to the second card.

Problem:
Since the analog data is generated on-demand at every 14 ms block, the program can't take any more than 14ms to generate the data and write it to the cards or else we will underflow the analog output.  Thus, any unwanted delays need to be avoided.  The data generation step doesn't take very long (in my testing it's turned off and taking < 1ms,) but I'm seeing some problems when I write the data to the PXI cards.  I'm using the DAQmxWriteRaw function to write the samples to the cards.  The function is being called twice every update loop: once for each output task (card).  When I added the second card, I noticed that occasionally the call to DAQmxWriteRaw would take much longer than 14ms, causing buffer underrun errors (-200016) on the second card, or regeneration errors (-200290) on the first card.  I inserted timing markers around each call to DAQmxWriteRaw, and I see that normally the calls each take around 4.8e-5 seconds to complete on card 1 (whether alone or in conjunction with card 2).  On card 2, the calls take usually take about 4.8e-5 seconds to complete, but they occasionally spike to between 5 and 40ms (e.g. 6 out of 300 blocks).

I'm wondering what would cause such erratic spikes in the time it takes for DAQmxWriteRaw to Ibe called on the second card.  Is there any known issue with using 2 PXI-6733s in the same PXI chassis?  Could there be some kind of resource contention (not enough DMA channels, etc?)  Both cards report that they're in DMA mode, and the FIFO fill priority is set to NotEmpty.  The problem also occurs when the transfer mechanism is set to interrupts.

Other Observations:
I noticed that the first timing spike corresponds to the point in the program where the "nicesru.dll" dll is loaded.  I can't find any information about which functionality is in this dll, and I'm not sure if it's a symptom or a cause.

I also noticed that once an error occurs, I can't seem to stop and clear the output tasks.  After trying to stop and clear all active tasks, the link light on the PXI chassis keeps blinking.  The only way to reset it is to go into the Measurement and Automation explorer and reset the second 6733 card manually (or use the ResetDevice C call).  It seems to be only the second PXI-6733, and it also happens when using the test panels in the MAX program.

Hardware:
PXI 1033 chassis with
1x PXI-6608 Timing Card
2x PXI-6733 Analog Output Cards
1x PXI-8461 CAN card
PXI PCIe-8361 I/F card

Dell Precision T7600
Xeon E5-2687W CPU

Software:
Windows XP SP3 32bit
NI-DAQmx 9.0.2
Visual Studio 2008 sp1

[rmattes]

I did a little more testing today. For reference:

PXI1Slot3: PXI-6733, First Card, about 5 years old.

PXI1Slot4: PXI-6733, Second Card, brand new.

Using either PXI card alone to write 6 channels of analog output works fine, the error only occurs when trying to use both PXI cards to write 16 channels of analog output. The delays are only happening on the second card. If I switch the order of the DAQmxWriteRaw calls, so that I'm writing to the second card first and the first card second, the timing spike still occurs when I write to the second card.

nicesru.dll seems to just contain functions related to string handling. It might be loaded when an error occurs to set the extended error information, but either way I don't think it's causing the problem.

Clearing tasks on the second card seems to fail all the time: even when using the test panel from MAX. I set the test panel to spit out an analog sine wave on one channel, hit start, and after hitting stop the LINK light on the PXI chassis continues to blink until I reset the card from MAX. Only the second card exhibits this behavior.

Given all of this, I think there may be some kind of resource limitation when using two PXI-6733 cards, or the brand new PXI-6733 card I have is broken in some way.

My next step is to try to upgrade the DAQmx drivers to 9.8.0 and retest. I'll also try flipping the physical order of the cards, so the new one is in slot 3 and the old one is in slot 4, to see if the problem is with which card is further down the PXI chassis, or if it's with the actual new card.

[cade]

Hello,

Can I ask how you are handling your DAQmx stop and clear functions on each iteration of the while loop.  Depending on how you are using them, this could be the source of the problem.

In the mean time, please let me know whether or not you see the same behavior after you download DAQmx 9.8.  Thanks

[rmattes]

Hi Cameron,

The problem with tasks not clearing was reprodcable in MAX: I pulled up the second card's test panel and sent analog output, and when I stopped it, the link light did not stop blinking until I reset the card.  My application does not try to stop or clear tasks while it is running, it only stops and clears tasks when it's finished. 

I got my DAQmx drivers upgraded to version 9.8.  This solved the problem with tasks not clearing on the second card.  It did not solve the problem with the delays.  I am still seeing that the second card's call to DAQmxWriteRaw is taking on the order of 15-30 milliseconds to complete, even with the timeout parameter set to 2 milliseconds.  I swapped the two cards' positions in the PXI chassis to make sure that one card was not broken, and I'm seeing that in both cases only the card in the second position is experiencing delays.  This leads me to believe that there is not a hardware issue with one of the cards. 

[cade]

Yes, I would tend to agree with you that something may be wrong with your card considering the circumstances.  Because the settings are the same for both cards and you are experiencing different behavior on each card, I think considering a repair may be a good next step.

You can call NI support at 866-275-6964 if you would like to start the process of getting an RMA for your board.  

[rmattes]

I don't think it's a problem with the card.  The card in slot 3 of the PXI chassis always works as expected, and the card in slot 4 of the PXI chassis always experiences large, varying, several millisecond long delays when calling DAQmxWriteRaw.  Switching the cards back and forth between the slots doesn't change anything, which is what leads me to believe that the problem is not with one of the cards.  I think the problem is somewhere in the DAQmx drivers, or has something to do with IRQ sharing, DMA channels or some other kind of resource contention.  Unfortunately I'm running out of ideas for how to troubleshoot those issues.

[rmattes · Accepted solution]

I was able to solve the problem by moving the PCIe card in the PC to a different slot.  I guess the problem was with how PCIe busses are managed by the motherboard, or how the resources were handed out to the cards in that particular slot.  Moving the card to the second 16x slot on the motherboard seems to have resolved my timing delays.

[cade]

Hello,

I apologize for misunderstanding you during my last reply.  I am, however, glad to hear that you aren't having the delays anymore.  Moving the PCIe card to a different slot was a good idea as the motherboard definitely could have been affecting the resources available at that particular slot.