tdm data does not load all channels

[JimNoel]

I have 12 files - each on identical in structure.  Each file contains 6 groups with identical channels of which are of different size.  If I load each file individually then all channels of each group are loaded.  However, after the 5th file is load, the sixth file and each thereafter does not load 2 text channels.  These 2 channels are txt compared to double for all other channels and half the channels can be near 1million points each

(Each file has a total of 35 channels so total channels are as follows 35, 70, 105, 140, 175, 208, 241, etc

Any reason this would be occuring?

[Brad_Turpin]

Hi Jim,

Nope, this should absolutely not happen.  Are you able to send us the 12 files?  If you change the order of file loading, does the trouble stay with the last few files loaded at the bottom of the Data Portal, or does it follow those files to the top of the Data Portal?

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[JimNoel]

No it seems to always happen on the 6th file that is loaded.  I tried a different of the 12 files to be the sixth and it always seems to remove the last 2 channels from that group.  This just started happening the last few weeks so not sure if it has anything to do with the file sizes getting larger but because it removed the data channels my report template gets all messed up due to the channel references being wrong.

The files are about 120MB each so it might take awhile but I will upload them to the ftp site labeled "6th dataset.zip".  I will post you when the upload is complete.

[JimNoel]

Brad

The file is uploaded to the ftp site.

[Brad_Turpin]

Hi JimNoel,

Both R&D and I were able to reproduce similar loading issues to those you reported when loading several of your TDM files at a time, and we were both able to load any one TDM file successfully.  R&D looked into this and determined that the 2 string channels with nearly 1 million values each were the source of the trouble.  String channels are stored in the TDM file in XML, which is why your TDM files were so big and not the usual 1 or 2 kB.  You have several options to sidestep this issue.  The easiest thing is just to not load those string channels or delete them immediately after loading them.  The best thing would be to change those two channels to enumeration channels that store an integer value for each possible string or state value-- this would be vastly more efficient.

A third option would be to change from the TDM file format to the TDMS file format, which stores strings inside the binary data file.  A major part of the problem with the string channels in the TDM file is parsing the string values out of the XML, and having the string values stored as binary strings in TDMS avoids this trouble.  I was able to register-load all 12 files as TDMS files, whereas I ran into an error when trying to register-load only 5 TDM files.  In general, with data files this large you should seriously consider right-clicking on your data file and using the "Register Data" context menu instead of loading the file outright.

What are you wanting to do with these 12 files?  Could you perhaps only load the particular channels from each file that you actually want to do something with?  Do you need to load the whole file?

Let me know how I can help further,

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[JimNoel]

Brad

Thanks for the information.  I will have to perform one of the methods you describe to resolve the issue and keep this behavior in mind for any future work.

However, is this issue going to be fixed in future versions of DIAdem regarding the limitation on channel size if defined as a string?

[Brad_Turpin]

Hi Jim,

The core issue will not change.  The TDM file format defines string channels as part of the TDM header file, using XML.  There is no intention to change this behavior, although that is theoretically possible.  The only reason the import scenario failed was that the XML parsing took up more RAM than was available on the computer for the DIAdem application.  DIAdem 11.1 and all previous versions are 32 applications, so even on a 64 bit operating system which can address 4 GB or 8 GB, etc., DIAdem can only use up to 2 GB of RAM.  DIAdem will eventually become a 64 bit application, and at that point it will be able to use the full addressable RAM that the operating system has available.  At that point your particular scenario would work-- let's say with 4 GB of RAM available to DIAdem.  But of course if your amount of string channel data doubled or quadrupled, there would come a point where you'd run into the same problem again.  You can always create more XML string channel data than you have addressable RAM if you work hard enough at it.

So the best solution would be to map those strings to integers and use an integer channel with string properties to store the enumeration states.  Numeric channel data supports random access in any binary file format, whereas string channel data supports random access in no file format.  The TDMS approach with string channels is a hybrid option.  The string channels still make the TDMS file much larger than it should be and much slower than it should be, but at least you sidestep the memory hogging and dog slow XML parsing step.

So in summary there are no plans to fix this particular behavior.  When DIAdem becomes a 64 bit application that will help a lot, but it will not solve the core issue.  The core issue is that millions of string values take up lots of space on disk and in DIAdem memory and load very slowly.  If your string data is simply the same 10 state values repeated differently row-by-row, then the most efficient approach is to use an integer channel to store one of the values {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} on each row, plus 10 string properties in that channel that list the corresponding string enumerations for each of the 10 integer values.

Brad Turpin

DIAdem Product Support Engineer

National Instruments