LabWindows/CVI Tip: Write Data to Disk Faster with TDMS Advanced API

[Anna_K.]

Contents

1. Overview of NI Technical Data Management Streaming (TDMS)
2. New TDMS Advanced Library
3. Comparing Standard and Advanced TDMS Functions
4. Guidelines for Using Advanced TDMS Functions
   1. Reading TDMS Data Asynchronously
   2. Writing TDMS Data Asynchronously
5. Reference List of New Advanced TDMS Functions
6. Where to Find Examples for the TDMS Advanced Library

Overview of NI Technical Data Management Streaming (TDMS)

The NI Technical Data Management Streaming (TDMS) file format is the fastest and most flexible way to store measurement data to disk. Developers have long used the LabWindows/CVI TDMS API to attach timing information and custom-defined attributes to measurement data as it streams to disk. Storing data in TDMS files eliminates the need to design and maintain custom data file formats while creating well documented data sets that can be easily searched and ported to any platform.

» Read an Introduction to the LabWindows/CVI TDM Streaming Library

Fig 4. Write measurement data to a TDMS file using the built-in libraries to better organize data sets by group and add descriptive meta-data about each data set.

Overview of the New TDMS Advanced Library

The TDMS Advanced API adds new functionality to the existing TDMS library, such as:

• Writing metadata and raw data separately
• (Windows) Reading and writing data asynchronously to increase streaming performance in some applications
• Overwriting a subset of data in a .tdms file
• Pre-allocating disk space for the data you want to write, which can prevent fragmentation and increase performance
• Add scaling information to channels in a .tdms file that will be applied when reading data from the file

Comparing Standard and Advanced TDMS Functions

Compared to the standard TDMS functions, the Advanced TDMS functions have the following advantages:

• Writing meta data and raw data separately—The standard TDMS  functions write meta data and raw data at the same time. The Advanced TDMS functions enable you to write meta data and raw data separately. You can use the TDMS_SetChannelInfo function to write the meta data into a .tdms file, and then use the TDMS_AdvancedAsyncWrite function to write the raw data into the .tdms file.

  Note  When you write data to a .tdms file, the data includes meta data and raw data. Refer to TDMS File Format Internal Structure for more information about meta data and raw data in a .tdms file.

• (Windows) Reading and writing data asynchronously—The standard TDMS functions only enable you to read or write data synchronously, which might result in lower performance when the buffer size is small and you have a large size of data to read or write. The Advanced TDMS  functions enable you to read or write data asynchronously. Use the TDMS Configure Asynchronous Reads or TDMS_ConfigureAsyncWrites function to allocate buffers and initiate multiple asynchronous reads or writes in the background.
• Overwriting existing data—Unlike the standard TDMS functions, the Advanced TDMS  functions enable you to overwrite a subset of data in a .tdms file. Use the TDMS Set Next Write Position function to specify the offset at which to overwrite the existing data.
• Reserving file size before writing data—Unlike the standard TDMS  functions, the Advanced TDMS functions enable you to pre-allocate disk space for the data you want to write, which can prevent fragmentation on a file-system level. Use the TDMS Reserve File Size function to pre-allocate disk space before writing data to a .tdms file.

Guidelines for Using Advanced TDMS Functions

Use the following guidelines when you read or write data with the Advanced TDMS functions.

Reading TDMS Data Asynchronously

The following guideline describes how to read data from a .tdms file asynchronously.

1. Use the TDMS Advanced Open function to open the .tdms file. Verify that the value of the enable asynchronous? is TRUE.
2. Use the TDMS Configure Asynchronous Reads function to configure settings and allocate buffers for asynchronous reads.
3. (Optional) Use the TDMS Set Next Read Position function to set the file position from which you want to start reading data.
4. Use the TDMS Start Asynchronous Reads function to start reading data to the buffers.
5. (Optional) Use the TDMS Get Asynchronous Read Status function to detect the number of buffers that contain available data.
6. Use the TDMS Advanced Asynchronous Read function to read data from the buffers.
7. (Optional) Use the TDMS Stop Asynchronous Reads function to stop reading data.
8. Use the TDMS Advanced Close function to close the .tdms file when the asynchronous reading process finishes.

Writing TDMS Data Asynchronously

The following guideline describes how to write data to a .tdms file asynchronously.

1. Use the TDMS Advanced Open function to open the .tdms file. Verify that the value of the enable asynchronous? is TRUE.
2. Use the TDMS Set Channel Information function to set channel information for the raw data.
3. (Optional) Use the TDMS Reserve File Size function to pre-allocate disk space for writing.
4. Use the TDMS Configure Asynchronous Writes function to configure settings and allocate buffers for asynchronous writes.
5. Use the TDMS Advanced Asynchronous Write function to stream data that you acquired to the .tdms file.
6. Use the TDMS Advanced Close function to close the .tdms file when the asynchronous writing process finishes.

Reference List of New Advanced TDMS Functions

Advanced Synchronous and Asynchronous I/O
Advanced Close File	TDMS_AdvancedCloseFile
Advanced Create File	TDMS_AdvancedCreateFile
Advanced Open File	TDMS_AdvancedOpenFile
Reserve File Size	TDMS_ReserveFileSize
Set Channel Info	TDMS_SetChannelInfo
Set Next Read Position	TDMS_SetNextReadPosition
Set Next Write Position	TDMS_SetNextWritePosition
Advanced Asynchronous I/O
Advanced Async Read	TDMS_AdvancedAsyncRead
Advanced Async Write	TDMS_AdvancedAsyncWrite
Configure Async Reads	TDMS_ConfigureAsyncReads
Configure Async Writes	TDMS_ConfigureAsyncWrites
Get Async Read Status	TDMS_GetAsyncReadStatus
Get Async Write Status	TDMS_GetAsyncWriteStatus
Advanced Synchronous I/O
Advanced Sync Read	TDMS_AdvancedSyncRead
Advanced Sync Write	TDMS_AdvancedSyncWrite
Advanced Data Scaling
Create Linear Scaling Info	TDMS_CreateLinearScalingInfo
Create Polynomial Scaling Info	TDMS_CreatePolynomialScalingInfo
Create Thermocouple Scaling Info	TDMS_CreateThermocoupleScalingInfo
Create RTD Scaling Info	TDMS_CreateRTDScalingInfo
Create Table Scaling Info	TDMS_CreateTableScalingInfo
Create Strain Scaling Info	TDMS_CreateStrainGageScalingInfo
Create Thermistor Scaling Info	TDMS_CreateThermistorScalingInfo
Create Reciprocal Scaling Info	TDMS_CreateReciprocalScalingInfo

Note  Use the standard TDMS functions when the advanced features are not necessary. Incorrect use of the advanced TDMS functions can result in a corrupt .tdms file. If you disable buffering when using the advanced TDMS functions, you must read and write data in multiples of the sector size of the hard disk.

Where to find Example Code for the TDMS Advanced Library

Did you find this tip useful? Rate this document or add a comment below.

If you give this a try, share your experience! Add a comment below.

[cymrieg]

Can I read data out of the file in one thread as another thread is writing data into the file?

I have a graph control in stripchart mode that is scrolling the data on the graph and to file. If the user scrolls back on the graph I want to be able to read data outof the file to put on the stripchart but at the same time incoming data from the instrument needs to be written to the file.

[Anna_K.]

Hi Cymrieg,

The scenario you proposed is supported and I have attached an example to the document above. In the code we have a thread that is writing some data to a tdms file using TDMS_AdvancedSyncWrite. In another thread we read a random chunk from previously written tdms data and do some basic verification to validate that read data. Be careful to change read position using TDMS_SetNextReadPosition before actually reading desired data.

Let me know if this is helpful!

Thanks

Anna

[cymrieg]

Anna,

Thanks for the response and this is good news so I will go ahead and play with the example.

What lead me to ask was that I saw in the API there was a function to "defrag" the file so I assumed you needed to do this between writing to the file and then reading from it, which of  course would have prevented writing and reading at the same time to the file.

So what is the "defrag" of the file all about? When would I need to do this?

[Anna_K.]

Hi Cymrieg,

Take a look at this document: KB: Why are my TDMS Files so Large? CVI uses the same defragment function as LabVIEW so the same use case applies. You might want to defragment your TDMS file if you are frequently writing small amounts of data for several channels.

Let me know if you have further questions.

Thanks, Anna