How to save record length in HWS files?

[ametcal]

Hi,

I have a program that streams Scope data to an HWS file (it works).  Now, I would like to load this data into Igor Pro for analysis.  Rather than play around with HDF5 loading stuff in Igor (though if anybody already has procedures to do this and don't mind sharing, I would be most appreciative to try them out!), I've decided to write a small LabVIEW program to read the HWS files and write ASCII text files.

In the ASCII file, I would like to put each record on its own line, so I can load a 2D matrix into my analysis program.  So my question:  Is there a way to programmatically save the record length (or "number of samples") to the HWS file from the data acquisition program and then load this variable into my data reader program to automatically split up the records on separate lines?  This would save needing to write down and enter my scope settings into the reader program each time.

Thanks for the help!

[DFGray]

The record length is stored by the underlying HDF5 file system as a property of the dataset, so the information is already there.  Igor Pro can read it directly, so it should not be too difficult, you just need to know how the file is put together.  Looking through the file with HDFView will make your job easier.

At the top level, you will find a group named wfm_group0.  This contains all your data.  There could be further indexed groups, as well.  Under this group. there will be four groups - id, axes, traces, and vectors.

id

This group contains identifying information for the data.  It can include such things as the operator, the instrument which produced the data, etc.

vectors

This group contains a series of groups calledvectorn each of which contain the data from a single waveform.  This is explicit data.  This is where you will find the data you are looking for, in a dataset called data.

axes

This group contains a series of groups called axisn. This subgroups contain the dependent and indepent variables needed to create a trace.  Each trace is composed of two axes (X and Y).  There are multiple types of axes, but the largest division is between explicit and implicit.  Explicit axes use actual data sets, and are usually Y data.  These are linked to a vector.  Implicit axes are calculated from a formula.  In the most common use case, this is a line and the data respresent the initial time and time increment.

traces

This group contains a series of groups called tracen.  Each of these represents a plotted line.  They contain links to the axes needed to plot the line.

Hopefully this helps.  Let us know if you have further questions.

[ametcal]

Thanks for the reply.  

Unfortunately, my HWS files are slightly different than you describe.  Maybe it has to do with versions?  I am using LabVIEW 8.6.1 and NI-SCOPE 3.8.7.  

More likely, it has to do with the way I am writing each record.  For my acquisition program, I started with the "niScope EX Save to File - HWS Low Level - Single Channel Stream.vi" example.  Then, because I want to stream 2 channels of data to a file, I put for loops around the niHWS New Wfm Reference and niHWS Write.  I only run the New Wfm References once, but loop over the Writes many times.  This puts the two channels in separate "axisN" groups in the HWS file, but they are both a single array that are really long with each record just concatenated next to each other.

I'm guessing that the way to split this up as you describe is to declare different groups for the different channels and then start a New Wfm Reference right before each Write?  My question is, will this cause any additional 'overhead' in the HWS file?  The way I currently finish writing the file is to save the scaling coefficents to each Wfm Reference before closing the file.  In my current configuration, this properly assigns the different scalings to the two axisN groups.  If I start saving separate groups for each record, will I need to set the scaling coefficients for each new group/waveform?  (depending on how many I save, this could potentially be a significant increase in file size)

Thanks again for the help.

[DFGray]

What you are doing now is the most efficient way to write the data to the file.  Almost anything you do will cause it to be more inefficient.  However, this extra inefficiency may not bother you.  Saving the scaling coefficients is only a few bytes, so should not bother you.  However, starting a new waveform will give you either a new trace or a full new waveform group (I cannot remember which), both of which are a lot heavier that just adding more data.  It will, however, give you the information you want and may not be that bad.  Try it and find out.

Note that what you really want to do is save your data in a 2D array.  Unfortunately, this is far beyond what NI-HWS was designed to do.  You really need a generic HDF5 API.  There is a really old one available here.  There are also quite a few newer ones if you search these forums and the web.