Binary Write vs TDMS Write

[adam_f]

I am wondering when it is better to write data to a binary file rather than TDMS file in real time.

I need to log 21 channels at 500Hz, 3 channels at 1500Hz, and 9 channels at 2Hz.

It was recommended to me that I should write to binary during data capture because of the fast rate.

Do you think there would be any problems writing this data to TDMS in rt?

Thank you.

[rramos]

Hi Adam,

TDMS is recommend by NI to streaming to disk. You can use it with no problem.

In addition, DAQmx provides an easy way to stream data to disk. Look this link: http://www.ni.com/white-paper/9574/en/

I hope this help.

Regards,

[crossrulz]

---

@adam_f wrote:

I am wondering when it is better to write data to a binary file rather than TDMS file in real time.

I need to log 21 channels at 500Hz, 3 channels at 1500Hz, and 9 channels at 2Hz.

It was recommended to me that I should write to binary during data capture because of the fast rate.

Do you think there would be any problems writing this data to TDMS in rt?

Thank you.

---

Because of the multiple data rates, I would recommend TDMS.  Each group in the TDMS file can be different sample rates.  And the Stream to TDMS in DAQmx is REALLY nice.

[adam_f]

When I try to write all these channels simultaneously in real time using TDMS my cpu load goes to 90% and memory usage steadily increases.

When I write to binary with all else the same, I hang out around 20% and have no problems.

I'm using a cRIO 9111 chassis with 9014 Controller (VxWorks).

Attached is a snapshot of some code.

Any thoughts?

[deppSu]

Data in TDMS files is structuredly organized. For fast access, TDMS do indexing while writing data which takes cpu and memory consumption. If you do care CPU and memory usage during logging you can try TDMS Advanced API http://zone.ni.com/reference/en-XX/help/371361K-01/lvconcepts/fileio_tdms_standard_advanced/, which has better performance but requires relatively more programming work.

[adam_f]

Hello everyone, thanks for replies, but still having the same issue.

I found a knowledge base article:

Memory Growth with TDMS Write 

http://digital.ni.com/public.nsf/allkb/961EDE096E98E87D862579ED0068BFF9?OpenDocument

This sounds like what is happening. 

I'm trying to implement solution #1 from that article which says:

1. If you are using the standard TDMS APIs to write data to multiple channels at a time, you must write the same number of data values to these channels in the same order.

Getting it to log that way is a challenge.

Does anyone have experience with this?

Any ideas how you could set up a program (for Solution #1 above) to log from multiple channels that are sampling at different rates?

[deppSu]

Another option is to try solution#3, you can try log channels of different rates to independent TDMS files and do manually merge to one TDMS file after the logging session completes.

[adam_f]

Thank you deppSu. That is probably the simplest solution and I did give it a try, but file size is critical for our application and that was getting too big too fast.

I haven't tried the Advanced TDMS yet as it wasn't listed as a solution for that problem.

I guess if the Advanced TDMS Write used a different caching algorithm than the regular TDMS Write it might make a difference with the memory growth?

[deppSu]

Yes, TDMS Advanced API can help reducing the memory growth in contineuous logging cases.