Problem with recorded number-of-sample when using multiple producer-consumer loops

[JCArifin]

Hello,

I am trying to simultaneously record signals from two sensors, QPD and Piezo.

To do this, I created 3 loops:

• 1st loop is to acquire QPD data (1st producer, datatype: array of waveforms)
• 2nd loop is to acquire Piezo data (2nd producer, datatype: float)
• 3rd loop  is to record and to write the QPD and Piezo data into a text file (2 consumers).

Problem is, when I tried to connect the 2nd producer to 2nd consumer in the 3rd loop, the vi recorded much less data than what I expect. I set 15 kHz as the sampling rate of the QPD. It works fine if I did not connect the Piezo producer to the Record loop; i.e., ~150k data in 10 sec (1st picture), but once I dequeue the data from the Piezo in the Record loop (2nd picture), I only got ~25k data in 10 sec. 

I tried to resample the QPD data into 1 kHz and 100 Hz, and the recorded data always much lower than what I expected if I create the 2nd consumer for the Piezo producer.

Could someone explain why this happens, and also advice me what I should do to get the expected number of sample?

Thank you very much

Kind regards,

Jane

Hardwares:

Piezo: NI USB GPIB H+ 

QPD Amplifier: NI PCIe-6351 with BNC-2110 

[crossrulz · Accepted solution]

Rule #2 of data flow: A loop cannot iterate until everything inside of it has completed.  Your consumer loop is being limited in rate by the slower of the 2 queues.  If the piezo only gives you 1 sample in a second but the QPD task is giving you 10ms of data, you will only dequeue at the 1 second rate.  And since you are flushing the QPD queue, you are essentially throwing away 90% of your data.  The solution is to just use 2 consumer loops: 1 loop per file.  And don't use the Flush Queue.

Additionally, most of your code in the sequence structure can be replaced with a couple of Build Path functions.

[JCArifin]

---

@crossrulz wrote:

....  The solution is to just use 2 consumer loops: 1 loop per file.  And don't use the Flush Queue.

.

---

So is it impossible to have the data written in the same text file? Because I would like to create a function of QPD signal with respect of Piezo position. So to achieve this, must I create a waveform of piezo data and compare its timestamp with the timestamp of QPD signal?

EDIT: Oh, and is there something wrong with Flush Queue?

[BertMcMahan · Accepted solution]

The problem is both sensors aren't being sampled at precisely the same time. How close in time are they, I don't know. You need to decide how accurately you want them to be sampled. If the QPD samples at 15 kHz and your piezo samples at 1 kHz, then you will have 15x as much data in your QPD as you do in your piezo.

Even if they're both at 15 kHz, you will experience drift over time if they're not hardware synchronized. For a few seconds or a few minutes of meausurement it's probably fine, but for a whole day you'll be out of sync.

The problem is both producers are producing asynchronously and your consumer is sampling synchronously. The consumer pulls one point out of each queue each iteration, then you have this bizarre flush queue in there.

Your data is disappearing because you're throwing it away with Flush Queue; if 100 samples get put into the QPD queue for every 1 sample the piezo puts in, you'll remove one sample from each, then Flush queue will throw away 99 of the QPD signals.

If one of your signals is much faster than the other, and you want to read at "max" rate, you'll need to decide which one must detect changes the best and which one can be delayed, then use timing from only *one* source to create your log.

You will have issues syncing things up since you're reading multiple samples at a time from your DAQmx function. While this is necessary, it means your data comes back in "chunks" and you'll have to compare timestamps to line up the data. You can do this programmatically using Align Waveforms in the Signal Processing -> Waveform Conditioning pallete. Use "Continuous" to do it as your data gets acquired and use "One-shot" if you do it after all your data is acquired. Note that this means you will need to change your piezo sampling code to generate a waveform, or you can do it manually.

[JCArifin]

Understood. I played around a bit based on what you said. And now it works as intended.

Thank you very much, crossrulz and BertMcMahan!