How do you stream data to a HDF5 file?

[SCJK]

I have managed to collect data to a hdf5 file but only for the first block of data (100, 1000, 10,000) but I would like to stream data until the hard disk is full or the user stops it. I have placed the HDF5 *.vi's, dll's, on my 'C' Drive.

[DFGray]

There are at least 4 HDF5 APIs available for LabVIEW.  Which one are you using?  If you downloaded the sfpFile API from the NI Website, the example I attached below (which also uses NI-SCOPE) should work for you.  You will probably need to relink to the appropriate API VIs when you load it.  If you do not have NI-SCOPE, the example will still give you the info you need.  Treat the scope VI as a generic data source and ignore the VI linking errors from NI-SCOPE VIs on load (you can get the NI-SCOPE driver here).

At each iteration, you need to do the following:

1. Get the current dataspace from the data on disk.
2. Extend this dataspace by the amount you will be adding (assumes an extensible data set, you can preallocate the size beforehand if you know what it is, which will make this step unnecessary).
   
3. Select a hyperslab on this extended dataspace corresponding to the data you will be writing.
4. Using the the new dataspace, write your data to disk.

This is a lot more complex than it could be, but that is the nature of HDF5 (very low level, very powerful).  Let us know if you need more information.  Please let us know what version of LabVIEW you are using, if you do.

[GWJ]

DFgray,

I'm just getting started with HDF5, and your streaming example should be most helpful. I managed to locate correct substitute VIs from the current API, except for two:  H5Sset_extent_simple.vi and DU64_2U32PlusU32.vi.  I can probably create the latter, but the former is mysterious. Can you direct me to a replacement for that? Looks like you working from an older version of the API...

-Gary 

[DFGray]

I am surprised the original poster did not run into this issue.  Attached is the original VI and its entire call chain.  You will need to have the HDF5 DLLs from sfpFile.exe installed on your machine for them to work.  Good luck.

[SCJK]

Thank you for getting back to me. I was trying to get a common file format (HDF5) which I could open files up in Labview and Matlab.  I have investigated this subject a little more but came across yet more problems.
I successfully wrote and read data using the Labview VI’s downloadeded from the HDF5 site and also successfully viewed the data in the HDF5 viewer program but for some reason Matlab was unable to read the HDF5 files produced by Labview and likewise the reverse.  I spent a lot of time on this and concluded that I should approach the problem from a different way, and just gave up!

[DFGray]

The version of HDF5 used by the National Instruments utilities is very old (1.4.4).  This version has a known bug which prevents it from working in some cases with newer versions of HDF5.  This is one of the reasons we have never updated it.

If you still need a solution, try TDMS.  It is native to LabVIEW, and we have a plug-in which allows Matlab to read it.

Message Edited by DFGray on 11-13-2009 08:13 AM

[SCJK]

Thanks for spending the time - yes we have already found this out.