Why is the FIFO overflow error 200141 unrecoverable?

[deserio@florida]

I have seen several threads regarding the FIFO overflow error 200141 during buffered event counting.  It seems that I and many others would like to see a way for DAQmx to ignore the error and continue.  As I understand it, on E-Series boards, the counter has a hold register (often referred to as a single point FIFO buffer).  The counter has hardware that can send the current count to the hold register whenever there is a gate transition and then it can signal that the hold register has new data.  This signaling is used to initiate a DMA or interrupt-driven transfer to a buffer for later processing of the data.  The 200141 seems to occur if a second gate transition occurs before the DMA or interrupt routine has had time to read the count strobed in on the prior transition.  Does the chip shutdown on this error?  What exactly sets off the 200141 error?  Is there a flag on the counter's staus register that says the last value in the hold register went unread?  Why can't the hardware/software read a second value to the hold regidster if the first went unread?   In my particular case, I continually count the internal 100 kHz clock and read it via DMA everytime the gate (connected to a bouncy switch) goes low.  I really only care to get the time count for the first gate transition and would simply throw away bounces due to glitching until a millisecond or so has passed. Then I would again start looking to time the next gate transition.  I am not interested in changing the hardware--I know how to debounce my switch.  I just would like to know why the error can't be ignored; why the task must be restarted.

[Jason_D]

deserio@florida,

The way the driver is made for this card is to ensure that you receive every sample and it does not have a built in feature allowing you to disregard the overflow error.  On cards with a larger buffer (S series) this is allowed as they have a larger buffer and there are more practical applications for overrating samples like this.  Most of the time there is not a legitimate need for this on these cards so one has not been included.  I have created a suggestion to R&D to let them know this is functionality you would like to have.  Here is my recommendation of another thing to try:

"Commit the task before the acquisition and then simply perform a finite acquisition repeatedly within a while loop.  You’ll need to start, read, and stop the task within the loop.  This will allow the hardware to capture the data as fast as possible using hardware timing, and the software to update the measurement on the screen after each iteration."

Let me know if this helps or if you have other questions.

[deserio@florida]

Jason,

It sounds like you are saying this was simply a choice on the part of the DAQ software engineers.

If it is a driver (software) issue, why isn't it sufficient to send the programmer the error,

but leave it up to him or her what to do with it, rather than forcing us to reprogam the counter?

As I said, me and others would choose other options for how to proceed

Regards,

Bob

[stilly32]

Bob,

When it comes down to it, the error is there to protect data integrity. The chip does not stop counting - when the FIFO over run condition is met the HW sets a flag for the driver and the driver immediately throws an error. The error prevents the driver from returning data with missing samples. If we ignored the error, we would have no way to tell you when the over run occurred or how often it occurred and would be giving you bad data.  We will always choose throwing an error rather than returning incorrect data. As far as just allowing a user to ignore the error, this is not possible with DAQmx. HW errors result in tasks being stopped.

In my opinion, rather than ignore valid errors in SW the best solution is to filter the input (which it sounds like you already know how to do) or use HW that supports higher rates. For your case, a bouncing switch is causing the problem and you'd want to throw that data out anyway - you can filter that signal with external or internal mechanisms and only acquire the data you need. As for higher rates, we have the exact same behavior in AI - we throw an FIFO overflow if the FIFO overflows and there is no way to mask/ignore that error. Even though you can acquire AI at up to 1.25 MS/s (on M-series) this isn't generally a problem because the FIFO on AI is ~4kS. The USB 621x device have a 1kS FIFO on counter inputs and they can buffer inputs at up to 1MS/s.   

Hope this clears things up, please post back if you have additional questions/feedback.

Thanks,

Andrew S

Multifunction DAQ Product Support Engineer

National Instruments

Message Edited by stilly32 on 12-11-2008 11:33 AM

[deserio@florida]

Andrew,

Thanks for clearing that up.  And thanks for passing on my suggestion.  I hope it gets implemented.  However, I did not suggest ingnoring the error flag on overruns and I am still not sure why NIDAQ can't both throw the error (via the read vi error cluster) and still return the data and not shut down.  How does this affect the integrity?  In both cases the user is alerted to the error.  I could almost understand the current logic if there were some information lost, for example, if the chip did stop transfers on the overrun error, but you seemed to say that the chip kept going---including DMA transfers I presume.  In that regard, does the driver throw the error immediately or does it find out about the overrun error when the user makes a  call to the read vi and NIDAQ reads the status flag?  NIDAQ would then be limited in any scenario (except via constant polling) to determining that the error occurred between the last two calls to the read.vi.

[Jason_D]

Bob,

You say "In both cases the user is alerted to the error.  I could almost understand the current logic if there were some information lost,"  When the overflow happens information is lost as it has no place to go.  I do not understand why you think nothing has been lost, the fact that chip doesn't stop counting doesn't mean no data is lost.

[deserio@florida]

Jason,

I should have said I could understand the present logic if any "additional" information might be lost by not stopping the acquisition.  Clearly on an overrun error, some unknown number of counts have already been lost, but what is gained by shutting down the acquisition process?  The question I meant to ask is:  Does stopping the acquisition on an overrun error provide any additional information that would not be available if the acquistion was allowed to continue?   I think you at NI have stated that the answer is no.   I am merely pointing out that, in fact, the reverse is true.  Shutting down the acquisition means any counts that could have been read after the overrun are permanently lost, and they would have been available if the acquisition were allowed to continue.  The software must, of course, inform the user of the overrun via the error cluster from the read vi.    But the decision of whether or not acquisition is allowed to continue should be left up to the user.   I guess it's easier to explain to a user that there was an overrun so we are not going to give you any more data.  It's a bit more difficult to explain that there was an overrun somewhere between this call to the read vi and the last one.  We don't know where in the data stream it occurred so it is up to you what you want to do with the following data. And then there would be the question of what to do if there are additional errors.  Say the DMA buffer overran as well.  Which error do you report?  Ah there are so many tradeoffs in data acquisition programming!

Regards,

Bob

[Kevin_Price]

I have to agree with Bob here that the user (programmer) should be given an option to continue the acquisition despite the lost samples.  Especially considering that 98% of the data acq boards with counter / timers have extremely small on-board FIFOs (1 or 2 samples) in the first place, which drastically increases the chance of producing such errors.

Side rant about "protecting users from themselves."  I get it.  I really do get it.  Most software products have to support users with a wide range of technical skills and interest, and in most cases, the majority of them fall on the less skill / less interest side.  And since the goal is to maximize users and profits, it's necessary to take pains to cater to the "basic" users.  Fine.  Makes sense.

Now for what I don't like -- when these design decisions become a significant impediment and/or pain in the neck to the more advanced users.  For example, at home I like to keep all my personal user-generated data on a separate network-attached hard drive.  So I have to struggle and fight with all my applications to try to convince them that NO, in fact, I do NOT want to store every single one of thousands of files directly into c:\Documents and Settings\<username>\My Documents.  I actually create, gasp, logical folder hierarchies of my own choosing.

The better apps make it fairly convenient to set up such a preference one time.  No problem there.  The not-so-better ones range from difficult to outright impossible.  (And even many of the better ones continue to excrete temporary files all over my temp folder without cleaning up after themselves. But I digress.  Ahem.)

Back to DAQmx now.  There are a number of legitimate applications that really don't care much about losing samples due to FIFO overflow.  I frequently include process monitoring in apps to keep a general eye on measurements and to make sure things don't get way out of whack.  So maybe once every few seconds or so, I query an encoder frequency task for the most recent 100 samples to see how things look.  I am often expecting a nominally constant speed, so I really don't particularly care if an occasional sample or two gets lost due to FIFO overflow.  If I got a FIFO overflow error in such an app, I would WANT to be able to say, ok, fine, you lost some of those previous 100.  No biggie.  Just give me the next 100 samples instead.

So the first attempted Read call would return an error and no data.  After that, the programmer should be able to call a function or (re)set a property node field that tells the task to continue buffering data while allowing it to be retrieved. 

Currently, the best workaround I know of is to stop and then restart the task.  I suppose that usually isn't *too* bad an option, but sometimes it can throw off overall system timing to then have to wait for the data acq buffer to fill up sufficiently for the needs of the app.  I too would like the ability to override FIFO overflow errors.

-Kevin P.

[stilly32]

I agree that the advanced user could make use of being able to ignore this error. I also appreciate that you understand the need to protect users from themselves - this alone makes this current behavior the correct behavior in my opinion, though I'll spare you the rant (it wouldn't be as good as Kevin's anyway) on supporting users where we failed to protect them from themselves    

I brought this up with one of our counter gurus (I'm more of an enthusiast with using our counters rather than knowing all of their inner workings) and he pointed out that while the current counters don't necessarily stop counting on error, we didn't explicitly design them to keep working after error so it may or may not keep working correctly by default. There would have to be a lot of testing done to just to see if it would work and even more to implement a setting and fully test it. We've focused more on adding features to help remove causes of the error altogether - PFI line filters, larger FIFOs on our newer 621x devices. Due to good feedback like this forum, we are aware that customers see this as a pain point. We are considering this feedback when working on future products and versions of DAQmx and I'm hopeful we will be giving you the performance/functionality required to meet your needs.

Thanks for the well articulated feedback and suggestions - this is heard and we do value it.

Andrew S