Best file format for variable multidimensional scientific data

[Tomi_Maila]

Hi,

I'm working on scientific data analysis tools. I'm handling and analysing scientific data of various forms and I'd like to use a common file format to store all the data. NI tdm file format would be quite ok, but for some reason the dimensionality of the data is limitied to 2D which is not acceptable. HDF5 is recommended as a file format. NI fileformat HWS is indeed using HDF5 as an underlying file format. Again, for some reason, the dimensionality of the data is limited to I think 2D, so HWS won't do. HDF5 DLL can be accessed from Labview directly, as is done in Soft Front Panel library. SFP uses old version of HDF5 and as far as I understood the HDF5 library is not thread safe. I'm afraid that considering multithreaded nature of labview, it would be too easy to create programs that would call the HDF5 library in non-thread-safe manner and cause data corrpution. Naturally data corruption is never an option.

Has anybody already struggled with the problem and found a suitable answer to the problem. What I need is a data format that is hierarchical like HDF5, thread safe, and allows storing meta data. The format must allow arrays of at least about ten dimension. It would be extremely nice if there was no limits to the number of levels of the hierarchy i.e. file format would allow trees of arbritrary depth.

EDIT: I'm using LV 8.0

Tomi

Message Edited by Tomi M on 02-28-2006 05:09 PM

Message Edited by Tomi M on 02-28-2006 05:11 PM

[DFGray]

I would recommend that you use the sfpFile HDF5 utility set, which you are already aware of (NI-HWS uses the same version of HDF5 - 1.4.4).  You will need to write functions to handle the multi-dimensional data you need, but you can use the 2D versions as examples.  You can serialize the access to the HDF5 DLL to prevent thread safety problems by using semaphores.  This will require a bit of discipline on your part, but is a fairly clean solution.  Semphores do take time to acquire, so use them only at the start and end of a full operation, not in every subVI.

I struggled with this problem several years ago and chose HDF5 as the solution, despite its flaws.  It is still the best solution for this type of problem that I am aware of.  HDF5 is supported by most major math/analysis packages, so it is a great choice for other reasons, as well.

Note that a conversion of the sfpFile utility set to HDF5 1.6.x is not trivial.  The calling interface changed in some subtle ways that did not break the C binary compatibility (much), but really broke the LabVIEW binary compatibility.  If you would like to know how to do it (and have a week or so of spare time) I can post the info. 

[Tomi_Maila]

Thanks DFGray, I'd appreciate if you could post the info. I can then consider if it is too laborous to make the transition to the present version of the HDF5 library.

Tomi

[Tomi_Maila]

I got information from HDF5 team that there is a way to compile HDF5 library using managed extensions to C++. This will propably mean that this way the library could be accessed using Labview .NET interface which is much cleaner than the DLL interfacing. DFGray, is the HDF5 DLL interfacing working sufficiently well that you think it is sufficient. NI folks, is .NET interface general and stable enough that it will propably do for interfacing HDF5 files.

Tomi

[unclebump]

Would Diadem work for this??

https://www.ni.com/en/shop/data-acquisition-and-control/application-software-for-data-acquisition-an...

Or maybe Citadel 5.

http://zone.ni.com/devzone/conceptd.nsf/webmain/725a6c3843f13c8786256ea600633724 

[DFGray]

DIAdem uses TDM under the hood, so it would not help.  Citadel may work, depending on the performance needed.  It has been awhile since I looked at Citadel.  Good suggestion.  The SQL connectivity makes it fairly usable from other apps.

If you want to use HDF5 1.6.x, you will need to write some "glue" code in C to interface it to LabVIEW.  The major problem is that in version 1.6.x, all the enumerated types went from constants to macros.  This works well in C, but LabVIEW cannot read DLL "constants", so there is no way to get to them.  The solution is a transition DLL which takes a LabVIEW enumerated type and translates it into the HDF5 enumerated type.  This was necessary for the data type specifier in 1.4.x, so you can see an example of this in the sfpFile code.  The C code is a simple switch statement for the translation.  I would guess there will be about a dozen to do.

You can read the release notes for changes from 1.4.x to 1.6.x, but the biggest other change I am aware of is to the link function.  It was expanded.  The old link function is now implemented with the new link2 function.  It should be relatively easy to fix the LabVIEW code by changing the call library function.  The HDF5 backwards compatibility method was a macro.  This won't work for LabVIEW, but you can use the template to rewrite the link function.  Or you can just create a link2 function and not use the old link.

Final suggestion.  If you are using LabVIEW 8.0, you may want to replace all the kludgy "cluster of two 32 bit integers" with a single 64-bit integer.  You can also rework the call library nodes to use 64-bit integers instead of doubles.  This technique was used because earlier versions of LabVIEW did not support 64-bit integers.

I have no experience using .COM, so cannot comment on how easy it would be to use.

Good luck.  Let us know how you made out.

[Tomi_Maila]

DFGray, do you mean by macro enums something like this

#define H5F_ACC_RDONLY    (H5CHECK 0x0000u)    /*absence of rdwr => rd-only */
#define H5F_ACC_RDWR    (H5CHECK 0x0001u)    /*open for read and write    */
#define H5F_ACC_TRUNC    (H5CHECK 0x0002u)    /*overwrite existing files   */
#define H5F_ACC_EXCL    (H5CHECK 0x0004u)    /*fail if file already exists*/
#define H5F_ACC_DEBUG    (H5CHECK 0x0008u)    /*print debug info         */
#define H5F_ACC_CREAT    (H5CHECK 0x0010u)    /*create non-existing files  */

The H5CHECK is a macro which defines a call to functions h5check(), which is not obligatory i.e. the call can be skipped. I have previously dealt with this kind of macro enums so that I have written a small VI which reads a file with the above #defines, extracts the names and the values as two arrays and writes these to a corresponding labview enum (or ring). Then I copied the enum (or ring) as a new strict typedef and used this as an input control for the dll.

If you DFG meant this, I think it's easy to handle. If you meant something else, I didn't quite understand.

[Tomi_Maila]

I came into conclusion that I'll use the library that came with SFP and think of extensions when I encounter limits. I will add support for selected higher dimensional arrays and maybe fix the U64 problem.

NI, are you planning to support HDF 5 natively, especially with arrays of 0-10 dimensions? There is a major lack in Labview at the moment that it doesn't support HDF5 and that tdm along with express VI:s in general only support 2D arrays!!! Every other major data-analysis environment in the world support HDF5 🙂 Perhaps it's too large a challenge for NI 😉

Cheers,

Tomi

Message Edited by Tomi M on 03-02-2006 11:42 AM

[DFGray]

Tomi,  you got it right.

[Tomi_Maila]

Referring to our old discussion about HDF5, I promised to tell you how I am doing with HDF5. I modified the NI HDF5 library to use native 64-bit references with Labview 8.0, added support for complex single, added support for files in memory and fixed some bugs I found. I could provide it here, but I don't know if NI is holding a license for it which restricts the distribution of the library.