cannot store complete internal data error

[aartjan]

Hello Diadem gurus,

I have some tdms files that I am post processing. I use diadem for that purpose. Some files cause the error "cannot store complete internal data error in the file."
See the attached picture. I assume it may be caused by my page file or a setting within Diadem. Any suggestions on how to prevent this?

Nasty detail is that the source files were actually lost! (I am saving changes made with diadem back into the same file, but in reality it simply overwrites the file.)

[Brad_Turpin]

Hi aartjan,

There were a number of improvements made to the TDMS DataPlugin, which debuted with DIAdem 10.1.  These improvements are available for download at www.ni.com/dataplugins for DIAdem 10.1 users.  However, there was one fix made that was too extensive to backport to the DIAdem 10.1 codebase, and this I believe is the one you need.  This last fix regarded the saving of enormous TDMS files.  The TDMS DataPlugin which shipped with DIAdem 10.1 expected to fit all the data values into memory at one time.  The TDMS DataPlugin which ships with DIAdem 10.2 Beta3 is able to buffer the data values to disk, therefore not requiring the entire file to fit in memory in order to succeed in exporting it to TDMS file.

The good news is that if you install DIAdem 10.2 Beta3 you will get the DIAdem 10.2 USI layer along with the correct TDMS DataPlugin.  Then even if you uninstall DIAdem 10.2 Beta3, the newer USI and TDMS DataPlugin will remain and work correctly with your DIAdem 10.1.  You can get the DIAdem 10.2 Beta3 by visiting the www.ni.com/beta website and requesting to participate in the DIAdem 10.2 Beta program.  Once you have requested this, remind me to activate your Beta profile, after which you will be able to download the installer.

Regards,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[aartjan]

Thanks Brad. I applied for the 10.2 beta and got approved. Next thing I read the press release of the release of 10.2. I might aswell wait for the official download..
After all, I have the service pack.
Aart-Jan

[CSchilling]

It seems like I have the same error when using the .mat conversion tool:

"Cannot store complete internal data in the file"____" with the format "MatLab". 

My file is 1.7 Gigabytes and I am using DiaDem version 10.2.0f3031 TDM

Is there a fix for this problem? A good work-around?

Thank you for the help. 

[Brad_Turpin]

Hi aartjan,

No, I'm afraid there's nothing we can do about this.  The MatLab DataPlugin uses the standard API from the MathWorks, which assumes that the whole file will fit in memory (RAM).  We were able to fix the TDMS file export because we control the source code of the DataPlugin, but we are simply reusing the MatLab file library and are unable to change its behavior.

If your data files contain more than 1 channel in them, you might try saving one channel per file instead-- you might be able to fit a single channel in RAM.

Brad Turpin DIAdem

Product Support Engineer

National Instruments