TDMS write time

[Slev1n]

Hey guys,

I have a snippet attached.

This snippet runs in a timed loop with maximum speed 500ms = 2Hz.Only the LIA values have always to be the most recent. The Analog and I2C values are not sampling that fast in another loop, so it is ok, if the last value is saved two or three times.

How do I have to set the timeout, that the loop ist still able to finish in time?

Any further suggestions how to improve performance?

kind regards

Slev1n

[Blokk]

Do you know how the Producer/Consumer pattern works? It is a lossless way to save data to file. I often use it in a way that the most important value which needs to be saved in a lossless way is acquired in the Producer loop. There I get some "less important" values (temperatures, etc.) from other parallel DAQ loops via the "Get Notifier Status", which gives the last uncancelled notification. I bundle all these data together in the Producer loop, and send it in a lossless way to the Consumer loop via a Queue.

As a quick fix however, you could replace the "Wait on notification" functions (which have timeout values set) with a "Get Notifier Status". In this way you use only ONE (the "LIA values" Notification ref) "Wait on notification" which will trigger when to write data to the TDMS file.

However, even I did not see your entire code, I think you have a bad design here. I do not understand why this logging loop is inside a timed loop??? You should really have a look at the Producer/Consumer pattern, the nice thing is that you do not care about how frequent the actual file logging happens, since you have a Queue. Also, it is silly to create a Time Stamp inside the logging loop, since the time information should be created when the data is created (by the hardware if it is available, or using Time Stamp in the Producer loop).

[Hooovahh]

I too suggest the producer consumer technique.  But with TDMS there is some more advanced function that could make life easier, if you take the time to understand them.  Under the TDMS advanced palette there are some asynchronous read and write functions.  The asynchronous write will ask to write some information to a TDMS file, but it will not wait for the data to actually be written before returning.  The result is the write function should take no more time then it does to request the write which I'd expect to be on the order of microseconds.  The data is then actually written to disk in parallel when ever the file can be written to.  I've personally never used this technique but there are some examples on the example finder.

[Blokk]

This is one possible solution when you have a single important measurement value which you want to save in a lossless way, and some other values like temperatures, which just need to be saved as most recent values...

Note that, I collect all DAQ values inside the producer loop, since I create there a common Time Stamp, so when you save all these values in the Consumer loop, it does not matter when it really happens... If you have a Time Stamp in the Consumer loop, you get fake Time Stamps in your TDMS file...

Take this example only as a sceleton, use proper TypeDefs and error handling in real application...

[Slev1n]

Thanks for your answer.

I know the producer consumer pattern. And it is applied here, too, but in a lossy way. I maybe mentioned falsely, that the LIA values have to be lossless. I use the notifier instead of the queues, because I want it lossy (I noticed the queue option "insert lossy" too late).

I acquire data from my FPGA and sent it to my RT system (myRIO) with different sample rates for different values (LIA, I2C, analog...).

Usually, the LIA values have the highest sample rate.

After some calculations on the RT the data is sent via WLAN to the host (tablet). Now I have different producer loops for the LIA, I2C and analog values which read the data (I use network stream here, only for I2C I use shared variable) on the host side.

It is written (on RT) and read (on host) with the same rate for each type of data.

Example:

Producer loops:

Read LIA values: 2Hz (display on frontpanel)

Read analog values: 2Hz (display on frontpanel)

Read I2C values: 1Hz (display on frontpanel)

Consumer loop:

Save data: 0,5Hz

So I have a higher samplerate for displaying than for saving. As I want to save only the most recent values I use the notifier. This should work, but I dont know how to handle this, if the "Save data rate = 2 Hz". Than the notifier would wait on the I2C value and the consumer loop would take too much time, therefor I wanted to use the timeout...but I think using "0" for the timeout is ok, as no data is saved for the first iteration of the consumer loop and if a value is in the notifier it can be read.

Regarding the timestamp: Every data (LIA, I2C...) acquired on the RT would have a different timestamp. Another problem are the LIA values which are averaged on the RT, each of them would have an own timestamp if every acquired value gets one. That is why I choose the way to get the most recent data of each on the host and save them with a timestamp. Than the saved data and the actual acquisition time do not differ much. The timestamp is only used to have a general idea what happened when and as the timestamp used has a resolution in seconds, it is fine enough 🙂

Could you maybe outline the benefits from using "get notifier status"? Also reading the help didnt give me an idea what is better using this function.

EDIT: Here is the Producer/consumer I use: Neglect the reference...dont know why the snipped shows them, actually there are local variables for the Stop button

[crossrulz · Accepted solution]

In these situations, I will typically use a global variable or the Current Value Table (CVT) library to hold the most recent data instead of the notifier.  You do not need to keep track of a reference that way nor do you need to close it.  Makes things a little easier.

[Blokk]

Could you maybe outline the benefits from using "get notifier status"? Also reading the help didnt give me an idea what is better using this function.

Ok, now I understand better what you want to do. So just take my advice what I wrote first. Use the "Get notifier Status". The advantage is here that, you just read out the most recent (latest) value from the given Notifier "channel". In this way you do not block/do not time your TDMS saving loop by the Notifications. Just run the TDMS loop as fast as you need, and read out the latest Notification values...

Edit: read crossrulz's advice above, and use a Global... and you do not care about Notifier references anymore...

edit2: hmm, interesting, I always learn something new by using this forum 🙂 I have never heard about CVTs before... Iit looks like it is just a kind of extended Functional Global Variable, isn't it?

[Slev1n]

Ok, now I see how you want to use the "get notifier status".

I will think about using the global variable as using variables is the easiest way for getting the most recent value.

And using the advanced TDMS stuff would be a good idea, but sometimes reading the explanations only confuses me even more.

Now I dont know whom to give the solution but at least you all get kudos 🙂

[Hooovahh]

---

@Blokk wrote:

I always learn something new by using this forum 🙂 I have never heard about CVTs before... Iit looks like it is just a kind of extended Functional Global Variable, isn't it?

---

Not quite.  It is most commonly wrapped in some kind of VIG, or action engine but it doesn't have to be.  A CVT in my mind is just a look up table of some kind.  A key / value pair that can be read and written.  I can write "Voltage" to be equal to 2.5, then I can read voltage anywhere else.  Of course if I write "Voltage" is equal to 3.0, then there is no history, and the previous data is lost.  You could be reading duplicate values (like reading 2.5 a bunch of times before 3.0 is written), or it might be writing 2.5, 2.6, 2.7. 2.8, 2.9, 3.0 and I only read at 2.5 and 3.0.  It is a lossy type of data storage, with all the pros and cons that come with that.

One of my favorite examples of a key / value pair wrapped in a VIG is the WORM (Write Once Read Many).  It is designed to write values into a functional global, that can be read back.  WORM throws an error if you try to write a value that has been written before, but it can be easily modified (and I have) to become write once, for specific keys.

http://forums.ni.com/t5/LabVIEW/WORM-Global/m-p/1226956#M521857

[Blokk]

Ok, clear, thanks!