How can I load Binary File in Diadem?

[Cap19]

I have a 1 GB Binary file having data of 8 Channels logged at 60KS/s. The Data type is a 2D array of double. I have created its TDM header file but when I load this File in Diadem, it takes 7-8 minutes to load. I am using Diadem 11.0. Is their any other way of loading Binary files to Diadem?

[Brad_Turpin]

Hi Cap19,

DIAdem provides four methods to deal with this scenario. The first one engages automatically when you drag&drop the file from the NAVIGATOR into the Data Portal-- DIAdem recognizes that some of the file will not fit into RAM/virtual memory and stores the rest of it in an internally managed HDF5 data layer. This is slow but limited only by the size of your hard drive, and you are able to edit the data values so loaded. Virtual memory is always slow.

The second option, "register loading" you have to ask for specifically. Instead of drag&drop, right-click on the data file in the NAVIGATOR and choose the menu "Register Data". This will use the data file in-place as DIAdem-managed virtual memory, instead of having the operating system or DIAdem make a copy of the data file somewhere else as virtual memory. This results in the data loading being almost instantaneous, independent of file size. Once a data file has been register-loaded, it shows up in the Data Portal with greyed-out channels that indicate that its values will be accessed on-demand from disk, and that they can not be changed in DIAdem. This is a particularly useful option for graphing ridiculously large data traces.

The third option to speed data loading requires you to put your data file into a Search Area or define Search Areas to include the data files in their current locations. This will result in the meta-data in those files being automatically indexed by the DataFinder. This leads to two excellent benefits in the NAVIGATOR. First, you can instantly see the full File >> Group >> Channel hierarchy of any file, without loading its data values, which enables you to select only the data channels you want to load or register-load. Second, you can select one of the channels at a time and see a curve preview in the lower right corner of the NAVIGATOR, without loading or register-loading the channel into the Data Portal. This is ideal if you want to look at the values of one channel in isolation. If you want to look at multiple channels on the same graph or plot ChY vs. ChX, then you need to load or register-load those data channels.

The fourth and final option to speed data loading is to right-click on the data file and select the menu "Data Reduction". This will pop up a menu with which you can load your selected data channels with 1:N data reduction. This divides each data channel into N sections and loads exactly 1 data point from each section. The simplest option is to load the first data point from each section, but you can alternatively load the minimum, maximum or average value from each section. It is also possible to load more than one of these values per section-- say both the minimum and maximum value from each section. This is slower than register-loading, but it has the advantage that you can change the loaded data values, because they get loaded into RAM/virtual memory.

Ask if you have further questions,

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Cap19]

Hi Brad!

I tried Register Loading, & it loaded the file instantly, but on performing any analysis in the data channels say FFT, it takes 8-9 minutes.

My RAM is of 2GB at 3.06 GHz.

[Cap19]

I am having Binary files of 1 GB each. One File contains data of 32 channels logged at 60KS/s and ther file contains data of one channel logged at the same rate.

Loading Time for both the files in Diadem was Approx. same .i.e 9 minutes. When I am performing FFT analysis on the first File (On all the 32 Channels at once) it takes 8-9 minutes, but on performing the FFT analysis on the second file which contains only one channel, after 10-15 minutes Diadem Hangs!

For loading a 1 GB tdms file in Diadem which contains data of one channel logged at 60KS/s it took 3-4 minutes, but on doing FFT analysis, Diadem was hanging!

Can you suggest, what will be the suitable file format in which files should be logged and what should be the size of file so that loading and processing of the data in Diadem becomes faster.

[Brad_Turpin]

Hi Cap19,

In general TDMS files are as fast as it gets for data reading into DIAdem.  1GB is a lot of data to read into an application, though I seem to remember it taking more like 2 or 3 minutes on my 3 year old laptop the last time I tried it.  It is possible that your TDMS file is severely fragmented, and that could be artificially inflating the load time.  To test this, try loading your 1GB TDMS file into DIAdem, then saving the TDMS file back out to a new file location.  Now compare the file sizes of the TDMS files.  If there essentially the same, then you can assume the TDMS file is not the bottle neck.  If they are very different, then try re-loading the new TDMS file and compare the loading time.  Let me know if you find differences and we can proceed along that avenue.

Assuming that the TDMS file is not the culprit, the only suggestions I have left are to run your calculations in batches.  The first thing to try is to load only 1 channel at a time from the TDMS file, calculate its FFT, then store the result to disk, delete the Data Portal, load the next raw channel from the TDMS file, calculate its FFT and store the result to disk, etc.  This sounds tedious, but it can be automated with VBScript to happen easily, and it may significantly improve the overall speed of your analysis.  A second level of batching could involve loading only a portion or time window from 1 channel and essentially calculating a time interval FFT, then proceeding on to the next time window from that channel, etc.  DIAdem can easily load a value index subset with the DataFileLoadRed() command, but if I were you I'd try processing all of 1 channel first and see if that doesn't improve things significantly.

Let me know what you find,

Brad Turpin
DIAdem Product Support Engineer
National Instruments