Retriggerable AI hanging and missing triggers on restart

[Phamton]

Hi, I'm trying to acquire a finite number of analog input samples (~1000) at a sample rate around 2 MHz after a trigger that runs at 100 Hz.  I'd like to do this for a certain number of triggers (what I call a "step" in my attached VI), change some external parameters and repeat.  I'd prefer to miss as few triggers as possible so I've modified the Multi-Function-Ctr Retrigg Pulse Train Generation for AI Sample Clock.vi example to commit the setup then start and stop the acquisition the tasks for each "step".  When I do this the acquisition seems to hang periodically.  That is, I can watch the trigger signal and the CTR0 output for the sample clock on a scope and see that the pulse trains occasionally do not trigger for long periods of time (10-20 seconds).  During this time I receive no errors.   Strangely the pulse trains do eventually start again.  Can anybody explain what's going on or how to fix it?  Is there something with the pulse train generation task that doesn't allow it to be stopped and restarted? 

Other observations:  Often when the pulse trains resume after not triggering the initial level is high instead of low.  I've also noticed that the delay between the trigger and the start of the pulse train seems to jump every once in a while by ~100 ns or so.  This isn't jitter (which I see as <50 ns or so) but discrete jumps in the delay.

I'm using LabView 7.1 with a PCI-6132 using DAQmx VIs.  I've attached a simplified version of my VI that exhibits the behavior.  Unsurprisingly the frequency of hanging seems to be proportional to the number of stops and starts (i.e. steps in my VI) so having a low number of triggers per step makes the problem occur more often.

Thanks,

Paul

Message Edited by Phamton on 10-30-2007 03:57 PM

[Phamton]

Sorry, attachment didn't seem to attach after I previewed my post.  Here it is.

[Hani R]

Hi Phamton,

I tried running your code on a PCI 6251 with no errors.  The case structure in the while loops may be causing some unnecessary delay on your machine.  Try removing the case structure and get rid of the DAQmx Is Task Done vi.  Specify the number of samples per channel on your DAQmx Read vi.  Also, make sure your trigger is TTL compliant.  Keep me posted on your results.

Message Edited by Hani R on 10-31-2007 05:21 PM

[Phamton]

Hi Hani,

 I thought that getting rid of the IsTaskDone? vi solved the problem.  I ran 25,000 stops and restarts without an issue but I just saw it hang again. 

  Note that it is not that my loop is only slowing down a little.  The pulse train literally stops and won't trigger for 10-20 seconds then starts up again.  The point at which it does this seems random.  My trigger signal is coming straight from a TTL compatible pulse generation box.

Paul

Message Edited by Phamton on 11-01-2007 10:27 AM

Message Edited by Phamton on 11-01-2007 10:30 AM

[Kevin_Price]

The only other thing I saw is probably not your main issue.  If you leave "triggers per step"set to the default value of 0, then your inner loop will stop immediately every time, and your outer loop that stops and re-starts the tasks will spin as fast as they possibly can.  I don't know of a specific reason why 1000's of stop/starts would eventually make the app hang, but it's the kind of thing I'd try to eliminate as a suspect.

-Kevin P.

[Phamton]

I've been running with a low number like 2 or 3 triggers per step so that's not the problem.  Hani, I also tried your other suggestions (although I wasn't sure I could eliminate the case statement) and they didn't work.  It does seem like the faster I make the inner loop the less problems I have (maybe why you didn't see any problems if you have a faster computer). 

It seems like something is going on in the hardware since the software continues to loop without error but the board doesn't trigger.  Are there any NI people out there that know what is going on in the board itself?  There can't be too many things that could cause it hang for 10-20 seconds then start working again.  Could something be getting messed up with the counters and they don't accept triggers until they roll over?  Are they any timeout parameters I could fool with?

I'd really like to get a handle on this problem as I'll be running my program continuously for days at a time and can't afford to have timestamps on my data that are intermittently 10-20 seconds off.  I suppose I could postprocess the data and find the hangups but I'd prefer not to.

Message Edited by Phamton on 11-02-2007 11:59 AM

[Hani R]

Hi Phamton,

I suspect the triggers are missed because your application is unable to update fast enough.  You can delete the case structure as long as you are sure to reconnect the necessary wires. What are the specifications of your machine?  What happens when you slow down the sample rate or reduce the number of steps (do you avoid the error completely)? 

[Phamton]

Hi Hani, thank you for your continued help.  My computer is running Windows XP.  It's a Dell Dimension 4550 Pentium 4, 2.4 GHz processor w/ 1 GB of RAM.  I'm still not sure what you mean by delete the case structure; if I do this how do I check to see if I should perform a read? 

  Regardless, I don't think that the computer speed could be causing my problems.  I don't see the software hang (the inner loop that checks for new data continues running) I just see the hardware stop responding to triggers for an indeterminate amount of time and then resume.  I ran a wire out of the inner loop from the iteration counter and found that on average the inner loop was taking about 25 us per loop to run.  So it's running plenty fast.

 I think there's something going on with stopping and restarting the pulse train task in the hardware.  I slowed down the triggering rate to 1 Hz and had the sampling rate at 1 kHz so I could clearly observe the individual pulse trains on a scope.  I noticed that the last pulse of the sample clock (CTR0 out) was being shortened when the inner loop finished.  It seems that the DAQ read occurs after the last rising edge and then the stop task immediately shuts off the counter.  When this happened the next pulse train would trigger but there was a delay of one extra sample clock before it started.  By stopping the task earlier in the pulse train I could see that the next pulse train was always delayed by the number of pulses that were cut off by the stop task.  I never observed it to not trigger for long periods of time with these parameters but I did see a variety of weird behaviors (delays from triggering, shortened pulse trains, not triggering, etc.) when I fooled around with the sampling rate and number of points. 

It seems pretty clear that if there are some issues with stopping and starting the counter task so for now I'm just going to reinitialize each time and lose some data.  However, if you know of any engineers with expertise on the counters I'd be interested to hear if there is a solution.

[Hani R]

Hi Phamton,

I'm unable to reproduce what you described in your post.  Keep in mind that the DAQmx Read is software timed.  Data will be pulled off of the hardware buffer when your computer is ready. Also, the last pulse in the train being shortened won't cause a problem because you are only concerned with the rising edge of the pulse.  I do have some suggestions for you. 

You can remove the case structure by right clicking the Case Structure and selecting Remove Case Structure.  Delete the Property Node as well (shown in the screen shot of the VI I edited).  Wire the Samples per Channel input to the Number of Samples per Channel of the DAQmx Read VI.

I also added a DAQmx Wait Until Done VI before the DAQmx Stop (also shown in the screenshot), which will ensure that the pulse generation is complete before executing the DAQmx Stop Task. 



Let me know if adding the DAQmx Wait Until Done VI changes anything. 


Message Edited by Hani R on 11-08-2007 03:18 PM

[Phamton]

Hi Hani,
  Thanks again.  Unfortunately the changes didn't help.  First I had to change the timeout value for the Read vi so when the hardware stopped triggering it wouldn't time out.  The Wait For Task Done vi always timed out waiting for the counter task to finish.  Were you able to get your hands on a PCI-6132 for the tests?
   I'm still a little confused as to the software vs. hardware question.  My understanding was that in my original VI the hardware acquires data and puts into a memory buffer (via DMA).  My loop checks inquires whether enough data has been collected to perform a read.  If there is enough it reads in the data from the memory buffer otherwise it loops back around to check for data again.  The hardware shouldn't have to wait for the Read in my inner loop, right?  Conversely, since I check to see if the data is there first, every time I perform a read it should immediately return.  So if I see my inner loop is running ~400 times before the data is there and my trigger rate is 100 Hz then my inner loop should be running plently fast, right?  I feel like I might be misunderstanding what you mean by software timed...
  With regard to the last sample clock pulse being shortened:  I understand the data is acquired on the rising edge so it shouldn't affect the analog input for that pulse train.  What I was trying to say was that it seemed like the counter task was being stopped before the last falling edge and I could clearly see that this affected the next pulse train even at super slow trigger and sample rates.  This leads me to believe that the counter task may not like being stopped and restarted.  Is there any documentation out there on the what's going at the hardware level in the counters in the mx vis?
 
  Another weird observation from today with a sample rate of 2.5 MHz, 100 Hz trigger rate, 1 trigger per step.  If I choose the number of samples to acquire (i.e. pulses in the pulse train) to be a multiple of 16 I have way fewer problems.  For example, 1024 samples works for thousands of triggers but 1025 samples stops triggering after only 5 or 6 triggers.

 


Message Edited by Phamton on 11-08-2007 05:20 PM