Need to improve speed when graphing large arrays

[minnellac]

Sorry, I'm going to be heading home for the weekend very soon, but the information you provided is helpful. I've found that even when there are bugs involved, there is usually a way to get what you want done in the end. So, if the bottom line is to prevent line mis-triggering on your acquisition, I'm pretty confident we can do that. There are a number of different ways to get the data into the computer, and I'm sure one of them will work. I'll take another look at Image Capture 13 on Monday, and write a simple data acq vi for an M series card that responds to an external trigger to acquire a fixed number of points. I'll leave the number of points, the trigger source, the sampling rate, and all the channel assignments configurable.

Moving the Start and Stop vi's out of the loop is a good idea, but there is some more configuration required to make the acquisition work the way you want. The line counter is just there to tell the vi where to put the lines in the larger array, right?

Now, if I am to understand correctly, the line count.JPG demonstrates a problem that has since been fixed by replacing the drivers. But there is still a problem with occasional missed lines, correct? I know it's a little odd to jump into the middle of this, but the problem is intriguing, and definitely seems solvable.

I'll get back to you next week.
Chris

[eweltmer]

Chris,

You are correct.  The problem appears to be corrected by the new drivers. The only problem we have left is the missing lines.

In looking at the problem we are missing a small but random  number of lines. Unfortunately, the missing lines ar also blank so when they are apparent, they are also noticable. Statistically, we are missing less than 20 lines out of 2000. This is less than 1% of the total. In talking with NI, the card was not made for this purpose & I tend to agree with them.

I think the a solution lies in  masking the problem by copying the previous line. I made a vi that does this after the image is stored but it takes a loneg time to add the entire array then replace the line: 5 min. with a P3:1 GHZ. I have tried to look at an array subset but this takes even longer.

Have a good weekend.

Earl

[minnellac]

Earl, I hope you had a nice weekend. Before we go nuts with modifications to the acquisition, can you try this simple modification? I ran an experiment with an M series card and a TTL square wave running at a high frequency (essentially, constantly triggering). I triggered an acquisition off a rising edge of the square wave (10 samples at 1000Hz = 10ms). When run in a loop like yours (Start and Stop in the loop) the acquisition took about 12ms on average. 10ms for the acquisition, ~2ms of overhead. By adding DAQmx Control Task before the loop, I was able to reduce the total acquisition time to 10ms -> the overhead was minimized. Can you try this (see picture) and see if it has any effect on the number of lines missed?

DAQmx Control Task is located in Measurement I/O -> DAQmx - Data Acquisition -> DAQmx Advanced Task Options.

While I agree that the M series cards weren't made for image acquisition, I really don't see why the card couldn't do what you want. 1000Hz sampling rate is pretty slow, when it comes down to it. The card doesn't have any problem acquiring that much data, or transferring it to memory. The trouble seems to be the overhead of the drivers. If we need to, we can eliminate that overhead all together. I think what Kevin was getting at was setting up your counters to perform the function of the sample clock and gating it on the digital trigger you are providing. The M series card would just be set up for continuous acquisition once using an external sample clock, and all the sampling could be controlled by your counters.

Can you tell me what the minimum and maximum expected time between triggers is?

Chris

[Kevin_Price]

I think what Kevin was getting at was...

Yep, that was exactly what I was getting at.  It sounded like there was a need to do Retriggerable AI with Finite Sampling on each trigger.  I knew the counters could be setup to generate the sampling clock and no DAQmx Start / Stop's would be needed inside the AI loop.

However, I didn't realize that simply "commit"-ing the task before the AI loop could make the overhead just about negligible.  Cheers for the DAQmx task state model!  Given the performance and the simplicity of the change, I'd heartily endorse your approach rather than messing around with counters.

As altenbach pointed out, there doesn't appear to be a good mechanism to sync the Digital data (from some type of external circuit) with the Analog data.  I *do* understand that the counter is counting a hw signal, apparently the same signal that should be the AI trigger.  However, there's still no control over whether the software call to Read the Digital Port is reading the count value that truly corresponds to the analog data being pulled out of the queue.

I suspect this lack of correlation contributes to the missing lines.   Let's suppose the lower loop gets bogged down on a particular iteration.  My top 2 candidates are "IMAQ ArrayToImage" and the "Contrast Brightness" vi, both of which are probably doing a lot more array handling than strictly necessary.  IMAQ is converting an entire 2d pixel array to an image rather than converting a single 1D row array.  Contr/Bright takes a 2d floating point input and produces a 2d integer output.  Unless very carefully constructed, the memory allocation functions are liable to get tickled pretty often.

Anyhow, let's suppose that the i_th iteration of the lower loop starts in sync with the i_th line of AI data.  However, the memory allocation functions have to do some extra work this time, so while the lower loop's i_th iteration continues, the upper loop completes both loops i+1 and i+2.  Next, the lower loop finishes and starts iteration i+1.  It gets the queued AI data representing line i+1, but when it does the Digital Read, it gets the "right now snapshot" data representing line i+2.  So AI data for line i+1 gets put into the 2d array as row i+2.  The lower loop now speeds around again since the Dequeue doesn't require any waiting.  It now reads AI data for line i+2 and again reads the Digital data representing line i+2.  So it overwrites the same row of the 2D array that it just got done overwriting the previous iteration.  This time however, it actually writes the data that belongs there.

However, row i+1 of the 2D array NEVER gets overwritten with AI data.  The initial 0 values are still there, hence blank lines.

An M-series board would let you perform hw-clocked, buffered digital reads.  With proper config and programming, you could guarantee to keep the AI and DI in sync.  Your need to hw-reset the external counter value tends to dictate for continuing to use an external counter circuit rather than using the onboard counters.  There IS a somewhat convoluted method that's pretty likely to work for an M-series counter, but it would be much nicer if the NI counters would support the simple type of reset you can easily accomplish with your chip.

-Kevin P.

[minnellac]

Good post Kevin. I was focusing on the top loop and hadn't really tried to figure out what is going on in the bottom loop. I am not as experienced with digital port reads, but here's a thought. Put the port read in the top loop with the AI read. Set the port read to occur on the same trigger as the AI read (can digital reads be triggered?). Now the line number and the data can be guaranteed to be in sync. Just bundle them together and pop them into the queue. Now it doesn't matter how quickly the bottom loop executes, the queue will buffer the data (as long as you don't run out of memory). Even if the port read can't be triggered, it can be wired to the error terminal from the AI read, but now it will have to execute serially.

This is assuming the lines don't come in in order. If they are in order, let's skip the port read all-together. There may be more overhead associated with the port read. If that is the case, it may slow down the AI read, and miss triggers (this could be happening now, and causing the slow execution of the bottom loop). It still seems like something that can be handled though.

Any thoughts,
Chris

[eweltmer]

Kevin & Chris,

First of all kudos to the both of you for taking the time to analyze the code & come up with solutions. This is much more that I had expected or anticipated.

Chris, I tried to put the DAQmx Control Task in the loop as you have suggested in a prior version but still had the alternating lines shown in my "Reply #60" Lines.JPG. But this was done many weeks ago before I came to the conclusion that we had a DAQmx driver problem.
I have included your suggestion & run the app with both drivers. The old DAQmx drivers still gives me that alternating line problems but the new one is much better. Attached is a picture: Scan 3 & Photo. Scan 3 has a few lines but photo is much better.
I need to revisit the different versions & techniques that I had used to resolve this problem now that I have the new drivers.

The minimum & maximum time between triggers is approach. 8 ms to 66 ms. As the period between triggers is relatively long, I have done some oversampling in previous versions then done some averaging techniques to remove the noise.

Kevin, I have tried to synchronize the top & bottom loops by moving the parallel port acquisition loop to the top loop then tying it to the AI trigger. Both bits of data was sent to the consumer loop using different formats: cluster & array. The general outcome at the time was that moving the parallel port read to the upper loop slowed the AI read & we had more skipped line. But this was before the new drivers.

I will try to tie the two acquisition & let you know.

What a difference new drivers & good programming suggestions make!

Again many thanks to the both of you.

Earl