Recovering TDMS files with DIAdem Connectivity Library

[AndreasK · Accepted solution]

If you are using DDC lib there is a way around the tdms~ file.

After your first trace (after you created all the channels and attached the first 100 points to every of them) call

bool afterFirstTrace(false);
// Create structure ...

while(measure) {
  // append measured data to channels ...
  if(afterFirstTrace) {
    DDC_SaveFile(filehandle);
    afterFirstTrace = true;
  }
}
DDC_CloseFile(filehandle);

After that the tdms~ should no longer grow instead tdms and tdms_index file should grow with every trace. After the DDC_Save the lib is in append mode and should work like you expected it.

Greetings

Andreas

[chasadams]

Append mode? Are you kidding me?? That is the Holy Grail. That is the missing piece of the puzzle for me. Thank you very much. You just solved all my problems with this software.

I was previously told that true streaming could not be achieved with this DLL. The way I was using the DLL previously resulted in ever increasing write times. It was explained that when appending new data to existing channels, the entire file reads into memory, the data is appended and the entire file is written back out to disk. This scenario comports with the time measurements I'd collected from testing. As the file grows, so does the time it takes to complete the write. It got so bad that it was taking more time to complete the write than time between write events and hence my application was thrashing. Understandably, this was intolerable.

Now, I'm concerned that such an arcane piece of information should have such game changing effects. What else lurks in there? What chance is there that I'm now taking advantage of a side-effect that might disappear with the next release? Why am I getting a log file full of exceptions? (This one might be answered in http://forums.ni.com/t5/DIAdem/Function-DDC-SaveFile-from-the-DIAdem-Connectivity-Library/m-p/718290...

Charles Adams

Lam Research

[chasadams]

No, I am not using DIAdem or any other software from NI except for the DIAdem Connectivity Library. I am attempting to create TDMS files with third-party software.

[AndreasK]

> What chance is there that I'm now taking advantage of a side-effect that might disappear with the next release?

This is no side effect. It is a problem that behavior that is related to write of tdms. The append does start when

the structure groups/channels is known. This happens after the first Save call. If *.tdm is used this is not necessary

to do the save call but please stay with tdms because it better fits your use case.

> Why am I getting a log file full of exceptions?

Are you still getting those log entries?

Do you know at which position in your code an additional line is added to the log?

Which version of ddc lib do you use?

[chasadams]

Yes, I still get a log full of exceptions.

This occurs the second time I append a block of data. There seems to be one exception per channel.

ERROR: TDS Exception in GetProperty: Tds Error: TdsErrObjectTransactionPending(-2530)

These exceptions are added to the file once and no other exceptions occur throughout the life of the file.

Examining my data, everything appears to be there -- first and second data and every subsequent data. If this is bad, it doesn't appear to affect the operation.

File version: 9.1.0.413

Product version: 2009

[chasadams]

I have a sneaking suspicion that once I put the file into Append Mode, I cannot adjust properties on groups or channels. At least, that's what I seem to observe. It looks like I'll have to close the file, reopen it and set what properties need setting.

[chasadams]

"I have a sneaking suspicion that once I put the file into Append Mode, I cannot adjust properties on groups or channels..." because I cannot save the file again. Error -6218: DDC_CouldNotSaveFile.

Writing a file property value is not the same as appending data to an existing channel, I suppose.

[DevM]

I am having the same problem with all the other user. I wonder if anyone can help me recover the corrupted TDM file that I have. Appreciate very much.

[StefanR]

Hi DevM,

the problem you observe might be different: You're TDM header has been corrupted by whatever kind of operation - I'm guessing while writing to TDM using DDC library, right?

In case you still have the information of how many channels has been written to the file and what data type the channels have (maybe you have the source code available), it might be possible to restore your data.

Greetings from rainy Aachen

Stefan

[AndreasK]

> Writing a file property value is not the same as appending data to an existing channel, I suppose.

I am able to update properties. Which property do you write?

How does your calling sequence look like?

Greetings

Andreas