Streaming from multiple devices fails if in certain slots

[wired]

My high-speed streaming project has been resumed, and I'm now under the gun to get it finished in 3 weeks.  Most recently, I was working on getting multiple PXI devices to acquire data simultaneously and stream it to a single file.  See the following post for all of that activity:

Using Non-DaqMX module as master to 4472s 

Now I am trying to acquire data from multiple PXI-6120 cards using multiple tasks.  I am generating a Sample Clock Timebase from a ZTEC GPS card in Slot 2, and putting it out on PXI_TRG2.  I am generating a Start Trigger from the GPS card and putting it out on PXI_TRG3.  I have a reentrant VI that sets up the task to acquire all 4 channels from a single device using the PXI_TRG2 and PXI_TRG3 signals.  The task also sets up an EveryNSamplesDaqIntoBuffer event.  I read the data out of the buffer in response to the event, and write it to a binary file.  The data acquisition VI is reentrant, and the Write DAQ Data to Binary File.vi is not reentrant.

Each instance of the PXI 6120 Sync and Stream One Device.vi is in a separate while loop that gets activated once all of the parameters are set up and the binary file has been created. A control reference to the binary file is passed to each instance of the VI.

With 2 while loops, each calling the data acq VI with a difference device's channel list, I am able to stream data to disk at an 800KHz sample rate (4 channels for each board).  However, this only works at this rate if my devices are in specific slots!!!  I can do Slots 5 and 7, 5 and 8, 6 and 7, or 6 and 8 (Slots 3 and 4 are currently occupied by PXI-4472 cards).  However, if I try to do slots 5 and 6 or slots 7 and 8, the software completely bogs down and I eventually get an error code of -200279 from each VI after reading the first block.  The user interface also becomes very jumpy when the data acq starts, which it doesn't do at all with the working slot combinations.  I can acquire data at a maximum of about 100KHz in the 'non-working slot' configurations, and even then I can see that the mouse is jumpy when I move it.  There is a huge performance difference between the 'good slots' and the 'bad slots'.  The fact that every individual slot seems to be a good slot (5 - 8 are all in the good configurations) makes it even more bewildering.

This makes absolutely no sense to me - the ONLY thing I am changing in the program is the device number (PXI1Slot5/ai0:3, PXI1Slot6/ai0:3).  Any advice would be appreciated!

Message Edited by wired on 03-13-2007 10:41 PM

[wired]

Nick,

I haven't had a chance yet to install the DaqMX 8.5 driver.  I'll need to round up a usb hard drive, since it's so damned huge it won't fit on my flash drive.  I'm putting out another fire tonight, and hope to get to it Thursday sometime.

Kevin

[wired]

Nick,

I took a break and installed DaqMX 8.5 and retried the application.  I have the exact same problem as before. 

I tried the memory check as you requested. With a 'good configuration', the CPU load jumped no higher than 23% during the data streaming, and the thread count went from 375 to 381 while the data acq was in progress. One task stopped just before the other, and I saw the thread count drop by 3, so there must be three threads being created per task.

With the 'bad configuration', the CPU load jumped to 100% when the start trigger fired, and stayed there until the daq tasks quite due to error. The thread count jumped from 375 to 381, and stayed there until the daq tasks were aborted due to error.  Seems like same number of threads are being created.

I hope you can get a DaqMX developer involved, at least so we can understand if DaqMX does something different based on slot.

Kevin

[Ed W]

Hi,

Thank you for posting to the NI forums.  There are a number of troubleshooting steps that we can take to find out why you’re having trouble with the synchronization.

1. Try swapping the cards around to see if the problems follow the modules or the slots in the chassis.
2. View the CPU usage as well as the memory usage during both the working combinations and the non-working combinations.  I would expect that one of those resources (probably the memory) is being overloaded, and that is what is causing the bad behavior.
3. For the file I/O portion of the code, make sure to open the file before writing to it (before the while loop) and close the file after the while loop.  Although this is probably unrelated to the synchronization problems you’re seeing, it may help to speed up your code.  If you don’t open the file before writing to it, the write to file VI may be opening and closing the file every iteration that it is called.  You may also try completely removing the file I/O portion and check to see if it makes any improvements.

Please post back with the results.  Thanks!

Ed W.

Applications Engineer

National Instruments

[wired]

Ed,

I tried switching modules, and have some head-scratching results to report.

The slot 5 and 7 combination was working properly, and slots 6 and 7 were not.

Test 1: Swap cards in slots 6 and 7, then repeat test of 5 and 7.  Results were good.  Still able to stream all 8 channels at 800K.

Test 2: Restore original configuration, then swap cards in slots 5 and 6, and repeat test of 5 and 7.  Strange results are as follows: The card in slot 5 took exactly 4 times as long to stream the data as the card in slot 7.  When finished, I checked the size of the streamed file, and it was the correct size for 800KHz, 8 channels, 10 seconds streaming, which is what I requested.  It's as if either the timebase divisor was different or the timebase for the sample clock was being divided by 4.  I received no errors from the slot 5 acq routine, indicating that it must have actually been sampling at 200KHz (otherwise there would have been a buffer overrun error).

My code is attached to an earlier post.  I don't see how this could happen.  The weird thing about it is that this occurred only by swapping PXI modules. Any insight into this phenomenon would be appreciated.

[Ed W]

Hi Kevin,

I agree, this is very interesting behavior.  Did you buy all of the DSA devices at the same time?  I’m just wondering if there was a major hardware revision between when you bought some of these DSA modules.  If you’re unsure, please post back with the serial numbers, and I can check our database to see if there were any major changes between releases.

Otherwise, if this doesn’t turn out to be the issue, then I can try to reproduce the issue here. 

Ed W.

Applications Engineer

National Instruments

[wired]

I tried taking out all the other boards, and leaving just 5 and 7.  Now the data acquisition works properly in those slots.  I'll try adding in boards to see what causes the failure.

[wired]

More testing.  I tried the following tests.  Here's the setup:

Chassis: PXI-1042Q
Slot 1: PXI-8186
Slot 2: ZTEC ZT1000PXI GPS card
Slot 3: Empty
Slot 4: Empty
Slot 5: PXI-6120 board S/N DAB7C0 (from sticker on main board) - call this Board A
Slot 6: PXI-6120 board S/N DAB7C4 (from sticker on main board) - call this board B
Slot 7: PXI-6120 board S/N DB660B (from sticker on main board) - call this board C
Slot 8: PXI-6120 board S/N DB6616 (from sticker on main board) - call this board D

Tests were run as follows:

Slots 5 & 6: Error -200279 both boards

Slots 5 & 7: OK

Slots 5 & 8: OK

Slots 6 & 8: OK

Slots 7 & 8: Error -200279 both boards

Slots 6 & 7: OK

With only Board A in slot 5 and board B in slot 6, got error -200279 both boards.

With only Board A in slot 5 and board C in slot 7, results were OK.

With only Board A in slot 7 and board C in slot 5, results were OK.

With only board A in slot 6 and board B in slot 7, results were OK.

With only board A in slot 7 and board B in slot 8, got error -200279 both boards.

With only board A in slot 7 and board C in slot 8, got error -200279 both boards.

With only Board A in slot 3 and board B in slot 4, results were OK.

[wired]

Latest tests: I removed the ZTEC card from the system, and reduced the data acq program to just start the two data acq tasks.  I changed the data acq tasks so that they use the internal clocks and no start trigger.  I still have the same problem.  I can, however, take data at 100KHz with each task with the boards in slots 5 and 6.  I do notice a delay in the GUI, as well as the event update rate - the event counter is not updating at constant intervals - looks like there is a slowdown followed by catching up.  Slots 5 and 7 work at all rates, but 5 and 6 do not.  I can also do Slot 5 at 800KHz and Slot6 at 100KHz, or Slot5 at 100KHz and Slot6 at 800KHz.  It causes sporadic mouse interruptions and uneven event update rates, but it does work. 

It's almost as if there's a different threading scheme or DMA configuration when the slot assignments change.  Is that possible? 

[Ed W]

Hi,

I’ve checked on all of your devices’ serial numbers, and there aren’t any apparent differences in their revision history.  There aren’t different threading schemes when the slot assignments change (or at least this is not expected behavior), but we are still looking into this.

Ed W.

Applications Engineer

National Instruments