Troubleshooting Error Code -200279

[BStevenson]

I’m working to develop an ‘auto-tester’ to verify operation of a device we test a lot of, and move away from testing by hand. I took over the project after the original developer left, and I have been stumped with an

intermittent Error -200279, “The application is not able to keep up with the hardware acquisition”. After receiving the error, the laptop running the application (connected via USB-C to a PXIe-1078 chassis via a PXIe-8301 card. A PXI-6224 and PXI-2564 card are also in the chassis) must be restarted or else errors about hardware errors occur at initialization.

The top loop in the block diagram samples 9 channels using a PXI-6224 DAQ card. That feeds the data to the lower three loops which save the data to a CSV, update indicators on the front panel, and drive the “sequence runner” that steps through the different test sequences. The sequence runner essentially closes relays on a 2564 card to provide power to/stimulate certain signals to the device, and then evaluates the output signals from the device (read by the 6224 card) against known good values to verify the

device is functioning correctly.

Everything I've read seems to suggest this error is caused by a memory/buffer overflow, potentially resolved by increasing the buffer size or decreasing the sampling rate. For the sake of debugging, I reduced the

sample rate to 10Hz and increased buffer size to 100000, but this does not seem to have helped.

Probably the most puzzling part is that sometimes, the program can execute 4 of the 6 sequences without issue before it throws -200279, but other times, it fails midway through the first sequence. I've even had days where I can run the program numerous times without any issues (program still fails mid-way through, as I'm still working out kinks in the sequences with how the signals are evaluated), but other days I can't even get one run in without a -200279. Given that the host laptop is being used exclusively for this application’s development right now (ie – no other major applications are using up resources), I would think the program would consistently fail around the same spot if a buffer overflow was the issue. Furthermore, I've

added indicators to show the number of samples in the buffer and to measure the loop duration that encompasses the DAQmx Read. There is consistently 0-1 samples in the buffer and a loop duration <5ms, including when the error occurs, which further confuses me as to if this is really a buffer overflow

type problem. 

I am admittedly not very well-versed in LabVIEW, so any thoughts/insights would be appreciated.

[mcduff]

Hard to debug a PDF even when you have the full version of Acrobat.

From your PDF it looks like you are reading a single point at a time. Read multiple points at a time. I typically multiply the sample rate by .1, that is read every 100 ms. Somewhat ironic, but that is 1 point for 10 Sa/s.

The other issue is that you have dependencies in your reading loop with those channel wires. I don't use channels and have no idea if they are waiting for data.

[BStevenson]

Sorry about the PDF, I'm not really sure if I can post the whole vi with company NDA/IP rules. I could probably provide the top level vi (pictured in the block diagram) if I scrub the indicator/variable names, if having that would be of value.

You're correct that I'm reading 9 channels at a time, 1 sample per channel. Are you saying you'd normally the 'NChan NSamp' instance and set it to read (sample rate * 0.1) samples per channel, which in my case happens to work out to 1 (which happens to be equivalent to using 'NChan 1Samp')?

I'll admit I'm not 100% sure how the channels work either, those were there when I got the project so they weren't my intentional design choice per se, but from what I can tell, they should always be able to accept data. And worse case that they're 'full' they just drop the oldest data they're holding, so I wouldn't think they're causing a 'backup' that interferes with the DAQmx Read. And if for some reason they were affecting it, I would have expected that my indicator for Available Samples per Channel would start to increase since the samples aren't being read from the buffer, but I have not observed that.

Given what I've read about the -200279 code, my thinking when troubleshooting this has been along the lines of "what could be happening between the DAQ card and the application, such that the application isn't reading from the buffer fast enough [causing the buffer to overflow]?" Do you think this is a good way of thinking about it, or do you think there could be something else that could be an issue but this is just the code that gets thrown?

[mcduff]

To make sure the card is working correctly without being affected by something else in your code, I would check out the DAQmx Examples included in LabVIEW.

Go to the Help, choose Find Examples and search DAQmx. Find the continuous input examples and test your card there. Once satisfied that it is working correctly, then start examining your code piece by piece. That is about the best I can offer without access to the code.

[Kevin_Price]

A partial shot in the dark here.

I'm also not a user of channels, but the channel reader in your daq loop sure doesn't look like the kind that would get stuck waiting.  Nothing else in the loop ought to either.

So then, how can a 10 Hz task with a 100000 sample buffer throw such an error?  Even if you never bothered to call DAQmx Read, it should take 10000 seconds (more than 2 1/2 hours!) to fill the buffer.

You also shouldn't be able to iterate the read loop every 5 msec in a 10 Hz task unless there was some big backlog prior to entering the loop.  But I see no data dependencies that would delay loop entry after task start, so that doesn't make sense either. 

   (For what it's worth, consistently finding 0 or 1 samples in the buffer makes *more* sense because your software loop *should* be able to keep up with a 10 Hz task.)

So that makes me wonder if something's wrong with the USB-C connection between laptop and PXI chassis?   For starters, it needs to be not merely USB-C but needs to be an official Thunderbolt port.   (Same connector, different top speeds, probably other differences.)

I've heard of a lot of people having finicky issues with Thunderbolt docks that sometimes require updates to BIOS, motherboard chipset, additional dock-related drivers, etc.   In short, my impression of Thunderbolt compatibility is that it isn't always automatic or fully consistent.

Try mcduff's advice first though for sure.  Don't suspect low level Thunderbolt issues before establishing that the shipping examples exhibit the same kind of failure.

-Kevin P

[BStevenson]

Sorry for the delay--apparently you can only post twice in the first 24 hours after creating your account. 

I ran the 'Voltage - Continuous Input' example, and set up the the number of samples to be 0.1 * sample rate based on mcduff's suggestion, with a sample rate of 100. I got it to run once without issue and was able to watch the signals change as I manually switched the relays to send power to the device under test. But the second time I ran the example vi, it failed almost immediately and threw the -200279 error. We have a spare 6224 card that I've tried switching out in case something was off on the card itself, but that didn't seem to help at all.

I also checked the port on the laptop I'm running from, and confirmed it has the thunderbolt icon next to the port where the cable to the chassis/8301 is connected. Are additional drivers needed to 'fully/correctly' (they're clearly communicating to a certain degree now) communicate with the hardware?

[mcduff]

Thunderbolt cables are also notorious.

What exactly are you switching? I ask because one time I was taking data by a big Marx Bank with a portable USB DAQ instrument. Every time the Marx Bank fired, the computer would think the USB cable was disconnected and I would get an error. Shielding the laptop and moving it further away solved the problem.

[Kevin_Price]

Some background for the OP:

The *normal* meaning of the -200279 error is that the device and driver are pushing data into the task buffer faster than your app is pulling it out.  When it's just about to write over top of old data your app hasn't yet read, that error gets thrown.  (Another way to say it is when the *backlog* of data becomes equal to the size of your task buffer, that error gets thrown.)

That is *not* the reason in your case.  The shipping example for continuous input can normally run for hours, or frankly, indefinitely without that error.  So now it's much more plausible to be suspicious of some combo of laptop drivers for Thunderbolt, DAQmx version or possible corruption, or the Thunderbolt cable itself per mcduff.

    And even so, it's a surprising error code to be thrown.  It says that the system is delivering too much data too quickly to your PC.  I'd tend to expect the opposite kind of problem -- not enough data (perhaps none at all) being delivered to your PC, resulting in a Read timeout.

Unfortunately, we aren't in the realm of normal troubleshooting any more.  This sounds like a pretty unique issue so I doubt I know what specific advice to give, other than to take heed of mcduff's further remarks about proximity to electrical or RF noise sources such as high current switching (or whatever a Marx Bank does... 😋).    I'm now also remembering a colleague who reported that when inducing 10's of kV static shocks to test products, one PC in the next lab over (but on the same circuit) would go blue-screen pretty repeatably while others didn't.

-Kevin P

[BStevenson]

The Thunderbolt cable came with the 8301 card and has National Instruments branding printed on the ends, so I'd like to hope it's not the cable. 

The relay is making/breaking connections between a rack-mountable power supply and the device under test via a PCB. Basically, we have a PCB with 4 connectors on it that the power supply, 6224 card, 2564 card, and device under test all connect to. The laptop, PXI chassis, connector PCB are all within 6-12in of each other, but the power supply is sitting 2-3 feet away. It is set to 22.5V and the device draws no more than 0.5A at any given time, so I think it is much less powerful than a Marx bank haha

[BStevenson]

Thanks for the background, I'm glad to hear that I'm not crazy for doubting the 'validity' of the -200279 error given what I've been able to deduce about data in the buffer.

The power supply involved is only set to 22.5V/device draws 0.5A max, so I'd like to hope it's not an electrical or RF noise issue. I guess I'll try to update/reinstall some drivers next week and see if that gets me anywhere. I'll look into the laptop thunderbolt driver and the DAQmx version and driver, are there any others you'd recommend? Also, I'm not very familiar with NI products, but can I just install/update to the latest version of DAQmx and its driver, or does the DAQmx version you need depend on your LabVIEW version (we have 2018)?