Varying Query Time for TDS 2024 B

[bcmasters81]

Hi everyone,

My Setup:

Tektronix TDS 2024B 4-Channel Digital Oscilloscope (USB connection)

Agilent 33250A Function Generator

LabView 8.6

My Problem:

I'd like to poll the TDS2024B once per second (1 Hz) and have it "Fetch Waveform" for all 4 channels.  The number of points is 2048 for each channel and the oscilloscope is triggered via Ext. Trigger port from the Sync Out of the Agilent function generator.

So, I've tried this, and added a small bit of code that measures the number of milliseconds since the last iteration of the While Loop, thus telling me how much time each loop takes to execute.  When I run this code, it typically takes about 850-860 milliseconds to execute, more than ample time to keep up with the 1 Hz (1000 ms) repetition rate.  However, once every 20 or so iterations, the loop takes something like 1100-1300 milliseconds to execute, which throws everything off, but then then the loop settles back to taking 850-860 seconds to execute again, until 20 or so iterations later.

I tried using the "Wait Until Next..." block within the While Loop, setting it to 1000 ms, in order to ensure that the loop takes the same amount of time each iteration, however when I do this, every 20 or so iterations, the loop takes around 1100 ms to execute, then the next iteration takes about 900 ms (obviously trying to sync back up with the 1000 ms requirement), and then back to 1000 ms per iteration.  And, as in the previous case, this repeats every 20 or so iterations.

I ran NiSpy on this to see if there was any clear indication of why this was occuring, and all I could tell was that, with good regularity, every 20 or so iterations, one of the four viQuery commands sent to the oscilloscope takes about 400 ms instead of the usual 100 ms (the four viQuery commands coming from the 4 channels I request fetching the waveform from, in sequence).  There doesn't seem to be any repeatability as to which channel on which this delay occurs;  the first time was CH4, then CH2, then CH3, then CH4, then CH1, then CH2, and after that I guessed it was going to be "random" across the 4 channels.

This can be pretty annoying because the goal of my code is to collect hundreds of thousands (100,000's) or more waveforms and save them to the computer as .csv files as I go, and with this delay, about 6% of my waveforms aren't even collected by the computer because the timing is out of sync.

I tried synching the oscilloscope and PC using NI-DAQ via a NI PCI-MIO-16E with a BNC breakout box; sending the trigger pulse from the function generator to the oscilloscope and the DAQ at the same tim.  I tried putting them in the same loop, with and without a 1000 ms timer, putting them in separate loops, with and without a 1000 ms timer, and pretty much every other permutation I can think of.  At the end of the day, I still can't get it to work.

The only thing that *does* work, is setting the number of points that I want for each waveform to a lower number.  Something like up 1040 points per channel seems to work, but I have no idea why this is, or why I can't get the full 2048, because the oscilloscope and PC are connected via USB 2.0, so there should be no issue with bandwidth.  In fact, Windows says that only 41% of the bandwidth is being used on that channel, so I have plenty of reserve.  You might say "well, use 1040 points or less per channel", but I can't because otherwise I miss about half the signal, and adjusting the time scale to make the entire waveform fit in roughly half the data points lowers the effective time resolution to unacceptable levels.  I can't do this at a rate slower than 1 Hz, because, as I said, I need on the order of 100,000 points or more, which, at 1 Hz, already takes over a full day to get, and I'm ramping up for experiments requiring 1,000,000 points, which takes 11 days as it is.  Cutting the rep. rate in half or in thirds would take 22-33 days for a single run, and there is no way to run my experiments that stabily for that long.

Does this make sense to anyone?  Shouldn't this system be able to handle the amount of data I'm trying to collect at the repetition rate I need?  The fact that the delay "glitch" occurs with some regularity and occurs regardless of if I used a 'Wait Until Next" makes me think I need to make some sort of timing or communication adjustment to make this work.

Any advice and/or guidance would be very much appreciated.

Thanks in advance!

[Joel-P]

Hello bcmasters81,

If you could post a snippet of the of the code that is interfacing with the TDS 2024 B, it would help me diagnose this issue. However based on what I have so far, I have a couple of ideas that we can check out.

First, you mentioned that the loop iterations typically take ~ 850ms and that the fetch commands take up roughly 4 x 100ms, what other processing is occuring in the loop that is taking the extra ~450ms? If you are writing data to a file in the loop, and that is taking up time, we could consider moving to a producer consumer architecture with a queue passing the waveform data.

My second idea is a generic troubleshooting step of checking this on alternate computers/DAQ cards/ anything so that we can limit the scope of what we have to investigate. Since this is using software timing there will be some associated jitter, although 300ms seems unreasonably long. There could be an issue of some other software running on the computer that causes LabVIEW to have a lower priority and slows the execution down. If that were the case, switching computers or attempting to disable any background tasks may help to alleviate the issue.

Thanks,

Joel

[bcmasters81]

Hi Joel,

Thanks for your response and for your ideas.  Since I'm using LV8.6, I've taken a screenshot, as I don't know if 8.6 supports snippets.

"First, you mentioned that the loop iterations typically take ~ 850ms and that the fetch commands take up roughly 4 x 100ms, what other processing is occuring in the loop that is taking the extra ~450ms? If you are writing data to a file in the loop, and that is taking up time, we could consider moving to a producer consumer architecture with a queue passing the waveform data."

Allow me to clarify, each of the 4 "Fetch"es consists of a Query, a Write, and 4 Reads.  There is 1 Fetch per channel, which then has these 6 sub-commands within it.  It is one of the 4 Reads within the single Fetch of one of the channels that takes 400+ ms over the usual 100 ms.  The 3 other reads for this Fetch and the other Fetches take less than 100 ms each.  No data is being written to a file in the loop at this time, just posted to a graph.  The ultimate goal would be to add data writing to the file, perhaps using a parallel loop to prevent what you mention, but at present, there is no other processing going on.

"My second idea is a generic troubleshooting step of checking this on alternate computers/DAQ cards/ anything so that we can limit the scope of what we have to investigate. Since this is using software timing there will be some associated jitter, although 300ms seems unreasonably long. There could be an issue of some other software running on the computer that causes LabVIEW to have a lower priority and slows the execution down. If that were the case, switching computers or attempting to disable any background tasks may help to alleviate the issue."

Unfortunately I can't do this since I don't have additional copies of LV8.6, and thus can't install on a 2nd computer as LV is already registered.  No other DAQ's to work with either.  However, my original message mentioned that this occured even without using a DAQ to trigger polling the scope, so I believe we can eliminate the DAQ as the issue.

I hope this clarifies things a bit.

Thanks,

bcmasters81

[Albert.Geven]

You could use an evaluation copy of LabVIEW to test on another computer.

The only drawback is a logo on your frontpanel.

[bcmasters81]

Unfortunately, where I work, "demo versions" are not allowed on our machines.

I have enclosed the NI Spy transcript for some of the communication between the PC and USB oscilloscope.  Most of the query/read/write calls occur with the same duration from one channel call to the next, however, #422 (viQueryf) takes 0.469 seconds, whereasthe other viQueryf's usually take 0.210 - 0.240 seconds.

Looking a level deeper, viQueryf has 1 viWrite, 4 viReads and 1 viPrint immediately after it.  The first viRead in this viQueryf is the one that takes the extra time in this case:  0.375 seconds rather than a more typical 0.125 seconds.

As I said before, the channel (1,2,3,4) on which this occurs seems to be random.  From this log, it happened on Channel 2.  This will repeat every 20-22 seconds or so by my count.

I am curious as to why the viRead takes *exactly* 3 times longer than usual.  Could this be a clue?

Any advice or guidance would be greatly appreciated.

Thanks,

bcmasters81

[Joel-P]

Hello bcmasters81,

Thanks for the additional information, I am still scratching my head as to why this error occurs in a periodic fashion and why the duration of this call takes what appears to be an integer number longer than usual.

Since we can't troubleshoot the PC or DAQ card, lets take a look at the call function. Could you run this code querying only 1,2,3 channels and see at what point the viread lags, and how many times longer the query takes?

This could be an issue with the way that the TDS 2024 B parses and communicates the curv? query, I downloaded the manual but was unable to find any technical details regarding the data communication. If you could submit a request for information/support from techtronix as well, it may be beneficial.

Thanks,

Joel

[bcmasters81]

Hi Joel,

I've tried what you suggest to see what would happen with fewer channels.  I randomly chose a channel (I flipped a coin a few times) and eliminated Channel 3.  Running 1,2 and 4 with 2048 pts each, worked just fine:  no periodic delay, or any delay for that matter.

So, I figured it might have something to do with the total number of points (3 x 2048 being small enough, but 4 x 2048 being too large).  So I re-wired the 3rd channel, so now running 1,2,3 and 4 at the same time, I tried various numbers of points per channel, but the same number of points for each of the 4 channels.  Here are the results:

4 Ch, 1040 pts/Ch -> 0 ms delay

4 Ch, 1100 pts/Ch -> 0 ms delay

4 Ch, 1200 pts/Ch -> 0 ms delay

4 Ch, 1300 pts/Ch -> 14 ms delay

4 Ch, 1400 pts/Ch -> 34 ms delay

4 Ch, 1500 pts/Ch -> 45 ms delay

4 Ch, 1600 pts/Ch -> 52 ms delay

4 Ch, 1800 pts/Ch -> 67 ms delay

4 Ch, 2000 pts/Ch -> 98 ms delay

The delay always occured just like before, once every 20 to 22 seconds...

So this is odd:  3 channels @ 2048 pts each is a total of 6144 pts total.  However, splitting this 4 ways across 4 channels would be 1536 pts per channel, and yet, even at 1300 pts per channel with 4 channels, there's a delay, as you can see above.

Let's not worry about the DAQ because I've removed it from the system and this problem still occurs, so we can rule it out.

I can try to contact Tektronix to ask about the curv? command and see how that works.

Thanks,

bcmasters81

[Joel-P]

Hello bcmasters81,

So it appears as though once we hit a specific amount of "query time" the delay is linear with roughly 0.115ms per sample.

This leads me to believe that we are running into some sort of bandwidth cap. I understand that the communication speed of USB should have more than enough room to comfortably fit all this data. However the curv? queries are either forcing a synchronization, getting hung up on a read/write, or have a finite speed limitation.

I think that talking this over with Tektronix will provide some much needed information regarding where this delay is coming from.

Thanks,

Joel

[bcmasters81]

Hi Joel,

Thnaks for your response.  Yes, it does sound like some sort of bandwidth issue.  I checked to make sure that I was using a USB 2.0 port rather than a 1.2, so as you mentioned, there should be ample bandwidth.

I realized that the limitation for points/channel isn't 2048, it's 2500.  I tried running 3 channels at 2500 points each, and it runs without any apparent delays at all (for 10000 seconds at least).  So your mention of the curv? command causing a synchronization issue, a read/write issue, or a speed limitation, does sound reasonable to me.  Is there any way to check for any of these?  For instance, the synch issue:  might there be some evidence of it in the NI Spy transcripts I've collected?  As far as the speed limitation, within XP there is a way through the Device Manager to see how much of the USB bandwidth is being used, and from what I've seen it's topping out at 41%, so there should be plenty of headspace there.  Any suggestions?

I posted the issue to the Tektronix online forum late last week, but no word yet.

Thanks,

bcmasters81

[Joel-P]

Hello bcmasters81,

I cannot think of another method to determine what is causing the error other than the data that we already have from NI Spy. The sync issue would likely appear as an inflated run time like we are already seeing. With regards to the speed of communication via USB, I know that we can increase the number of channels that are used when we are using DAQmx via a property node, however I am unsure of how to port this functionality to the device driver VI's that you are currently using. I will look into implementing this into the drivers, and I should be able to get back to you in the next couple of days.

As it now stands, would we be able to maximize the number of data points collected to create a delay that is just sufficient to not lose a waveform, and would that be an acceptable temporary solution?  I understand that the time resolution will decrease, but based on the regression of delay time based on data points, does this reduce the resolution to unacceptable levels?

Thanks,

Joel