Better than polling for a full buffer?

[AlecSt]

Hi all,

For background: I am wiritng a DAQ VI with a producer consumer model, strongly based on the Continuous Acquisition and Logging VI.

I am writing the acquired data to a buffer. When the buffer is full, I want to write that data to a file -- in other words, tell the "logging" loop to start writitng.

I would like to know the best way to do this. Right now I am doing the following thing: polling for a full buffer, and then if full, enqueuing a command for the logging loop to start writing the data to file.

[mikeporter]

Why does the logging loop need to be told anything? It has access to the buffer, right? Why can't it decide on its own that it's time to start saving data?

Mike...

[AlecSt]

That's a good point. I should have mentioned that I'm trying to minimize the number of writes-to-file. There might be something silly about what I'm doing, so please tell me if this is inefficient:

[mikeporter]

OK, that doesn't really change the questions. Somebody has to decide when to write to the file using some criteria that is likely to change over time. Seems to me that the most reasonable place to put that logic is in the loop that is going to be writing the data.

Mike...

[AlecSt]

So should one poll for a full buffer before writing to file, then? Or is there a better way? I feel like polling for anything is generally inelegant.

[Blokk]

The point is that, your OS does buffered file IO. Even if you are writing to your file at every iteration of a while loop (lets say at 1 Hz), the OS does not write in real that often: OS writes to your file in chuncks. So I would not worry about buffering. Just put your data into a Queue, and at the consumer loop write to file. What is the DAQ rate? If you do fast acquisition, you might want to have a look how to use DAQmx TDMS operations, which is very fast. Either the case, you do not need to explicitly handle buffering...

[crossrulz]

A little more context would be nice here.  What file format are you using?  Where is the data coming from?  Where/what is this buffer you are referring to?

[smithd]

Using DAQmx built-in logging is probably better. If that doesn't fit, you're doing premature optimization. You only need to focus on write size if you need crazy high performance and then you start having to worry about buffering and the like. Since labview will buffer for you and the os will buffer for you, you can just leave it be until you've identified that loop as the bottleneck. 

Also, this is a peeve of mine so I'm going to call it out specifically:

"I am writing the acquired data to a buffer. When the buffer is full, I want to write that data to a file -- in other words, tell the "logging" loop to start writitng."

The logging loop should know, itself, when to log. After all, it only has that one job. Is that job really so hard? It just sits there, waits for data, and shoves it in a file. 🙂

My point is, *if* your loop needs buffering to hit the write-to-disk performance you need, that buffering is the responsibility of the logger loop not the producer loop. That way the producer has just one job -- producing -- and the logger has just one job -- logging. Any other configuration can and will lead to suffering, eventually.

So to answer your question rather than giving the super obnoxious "don't optimize" statement above, I would say you leave your producer just writing to the queue. On the receiving side you have an allocated array of data and a counter. Every time you receive a packet from the producer you replace some subset of your buffer array and increment your counter. When the counter reaches N, reset the counter and write your data to disk. This keeps everything event-based (reading from the queue) and also eliminates the pesky problem of having the producer doing the logger's job.

[Yamaeda]

---

@AlecSt wrote:

Hi all,

I am writing the acquired data to a buffer. When the buffer is full, I want to write that data to a file -- in other words, tell the "logging" loop to start writitng.

I would like to know the best way to do this. Right now I am doing the following thing: polling for a full buffer, and then if full, enqueuing a command for the logging loop to start writing the data to file.

---

So, assuming the buffer is a queue or AE in it's own consumer state, it can queue a write-command to the consumer with the data and clear its buffer.

/Y

[AlecSt]

The context: I'm taking data with a high speed digitizer and writing to binary files that we can read with our C++ analysis software. I'm exploring the option of using TDMS but only after we are up and running with the current setup.

The digitizer can sample at 2ns and we take around 1000 samples for event. So in the extreme case we would need to write every 2 us or so, on average.