DDC_AppendDataValueFloat takes longer each use

[chasadams]

I hope this is the right place to ask this question...

I'm using the DLL in third party tools (my tools) to write several hundred data points to several hundred channels every 10 seconds. Each successive write (DDC_AppendDataValueFloat) takes longer. This continues until the time it takes to write this data exceeds the 10 seconds used to gather the next batch of data. At this point, it starts thrashing. Actually, well before this point, the computer is unusable.

This operation needs to take no longer than 200 ms -- less if possible. In the beginning, the times are in the 150 ms range.

I've isolated and timed the call I make to this interface and can come to no conclusion other than: that's how long it takes to write the data. It is as if it must seek the end of each channel to append the new data. Surely, I'm wrong. Any help out there? Do I need to give more details?

[TaylorG]

Hi chasadams,

Are you using DiaDEM with this?  Are you running any other processes while running this?  Also, what DLL has this DDC_AppendDataValueFloat?  Is this a DLL you wrote?  Also, does the computer actually crash, or does it simply begin to lock up?

Perhaps there is a memory leak somewhere that is causing these resources to be used up and causing the delay for consecutive writes.  Some more information about your program would be very helpful in getting to the bottom of this.  Thanks, chasadams!  Have a great day!

[chasadams]

I am using the DIAdem Connectivity Library, a suite of DLL files. DDC_AppendDataValueFloat is one of the many functions in this library. My application makes calls into this library to create and populate a DIAdem file of type TDMS.

So, I am not using DIAdem, per se, nor am I using any other NI product -- just this library.

I'm running on Windows so, yes, there are plenty of other process running. I'm not prepared to vouch for them all 🙂

This problem causes my application to thrash. It is capable of running out of the memory allotted by the OS, and subsequently dying.

I'm prepared to accept that I'm still making some error in the way I use the library, but as I said I've isolated the problem to this one function call. Perhaps someone with experience using this library can shed some light...

[TaylorG]

Hi chasadams,

How large is the data that you are trying to append each time?  Is the data always the same size?  Also, how long can you run the application before this happens?  If the data is rather large, writing several hundred data points several hundred times could be causing these memory errors.  Thanks!  Have a great weekend!

[Brad_Turpin · Accepted solution]

Hi chasadams,

The DDC library does not stream to disk.  If you ask it to append to the end of a TDM file, it will first load the current contents of the TDM file into memory, do the appending in memory, then save the expanded contents back to a new TDM file with the same name as the original.  If you ask the DDC library to append to the end of a TDMS file, the same process occurs.

This is the reason that NI created the TDMS API (TDM for Streaming), but that only ships with NI products (LabVIEW, CVI, MeasurementStudio).

You could alternatively stream to binary file(s) and use the TDM Header Writer DLL to create a matching TDM header file for your binary data file(s).  This is also a free DLL that you can download from NI's web site.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[chasadams]

Brad,

Thank you. That sounds like the answer I've been looking for. This distinction between the DDC library and the TDMS API product is key and I will acquire the latter as soon as possible.

Charles Adams

Rudolph Technologies, Inc.