MYSTERY: Buffered event counter values on multiple synchronized counters are not the same even when they receive identical inputs

[cwierzynski]

Dear Forums

I am using a PXI-6608 to capture the times of TTL pulses and acquiring these times as 'buffered events.'

Since I have 3 sources of TTL pulses, I am using three counters at once. They have the same start trigger so that they are synchronized. The Source of all the counters is the onboard 10MHz high-accuracy oscillator.

To test the setup, I am feeding the identical TTL pulse train to the Gates of all three counters. The pulse train is around 30Hz. I expect the values of the counters to be identical at every rising edge of the pulse train.

But...they aren't. Most (99.1%) of the time, they are identical. And sometimes (0.87%) one counter will be off by 1 tick. BUT every once in a while (0.03%), one of them will be off by 25 clock ticks. And not always the same one.

I am using interrupts to retrieve the event values, not DMA, although not sure why this would matter.

I get the same results regardless of whether I use the 6608's filter to debounce the inputs.

Any clues as to what's going on with this highly mysterious situation would be greatly appreciated!

Thanks,
Cas

[gus]

Can you post the code you are using to reproduce this?  Are you physically wiring the same input to three different counters, or do you have them all using the same PFI line for their measurements?

[cwierzynski]

Hi Gus - thanks for your reply.

I am physically wiring the same input to three different counters.

Unfortunately I can't easily disentangle the counter code from the rest of the app.

But, if it helps, I modeled it very closely after the Count Buffered Edges (NI-TIO).vi in the NI Examples. The only differences are

1. I'm using three counters, and routing RTSI4 to all of their start triggers. (RTSI4 is carrying the AI Start signal from another card on the chassis).

2. I'm routing the 10MHz OCXO to RTSI2, and specifying RTSI2 as the source of the three counters.

It seems like the three counters are starting at the same time and they aren't drifting relative to each other over many days. The problem is that every once in a while, at random times, a random counter will report a value that's 25 ticks different from the others.

It seems unlikely that it would be a synchronization issue, because 25 ticks (2.5 usec) seems like a long time.

Any tips are greatly appreciated!

Thanks,
Cas

[gus]

This is really strange.  I've never heard of a scenario like this one. Which three counters are you using?  Have you tried switching between the different counters?  You might want to try just selecting the same input (GATE) pin for each counter.  That will eliminate possible issues related to physically wiring the same signal to three locations.  My next suggestion would be to try using another counter on the 6608 to generate the start trigger to try to isolate your issue further.  And one more question, what type of filtering did you use?

gus....

[cwierzynski]

Hi Gus - thanks for your suggestions, all very useful. I am going to try to put together a simple test vi that can reproduce the problem so that I can share it more widely -- I will incorporate your suggestions into this test vi.

I am using counters 4, 5, and 6 (starting with 0), so they are on the same ASIC. I haven't tried other counters.

I get the same result using no filtering and using 5usec filtering. I immediately suspected something funny with the 5usec filtering when I saw 2.5usec glitches, but turning off input filtering gave me the same results.

Cas

[cwierzynski]

So I tried your suggestion of having the three counters read from the same input pins. That eliminates the 25 tick differences. There are still occasional (about 1% of the time) differences of 1 clock tick, but I can understand that as skew inside the board.

I tried simplifying the wiring, using only two counters instead of three and making the two wiring paths short and as symmetric as possible. I still get rare 25 tick discrepancies, though.

Another clue is that sometimes the 25 tick discrepancy is negative - ie one counter is gated ahead of the others!

Cas

[Thaison_V]

Hi Cas,

I hope you're doing well.  It sounds like having all three counters use the same PFI line helped solve the 25 tick differences, but you'll still get the 1 tick difference every so often.  Is this with the filter enabled on the PFI line for the gate?  Did you have a chance to try this on a different set of counters as well?  This doesn't completely explain the disappearance of the 25 tick discrepency, but it would be interested to find out as well.  If you continue having trouble with your application, we can try reproducing the issue here if we can get some type of reproducable code from you, so definitely keep us posted!

Thaison V
Applications Engineer
National Instruments

[cwierzynski]

Just thought I'd "close" this ticket, since I found the problem. Turns out the problem was an every-so-slightly flaky signal generator that I was using to generate the TTL pulse train. For some reason, on the order of one pulse out of 10,000, on average, was not within TTL spec and would be read differently at each input.

When I use a good, reliable pulse generator (a nice M series card from NI), the "glitch" goes away. There are still 1 tick differences sometimes but this makes sense.

Thanks to all who gave me great debugging advice on this one.

Casimir