On-Demand DataPlugin

[kdaily]

I am trying to create a DataPlugin that will read a zip file that contains zipped binary files. A coworker created a One-Shot DataPlugin that reads and displays the data from all binary files but we want to be able to read and display (in DIAdem) only the selected binary files’ data that a user is interested in. This seems like we need to create an On-Demand DataPlugin but from examples I have found I can’t find how to implement the user selection.

[AndreasK]

As far as I know there is no user interaction in the plugin.

What you can do is "Selective Opening". (Right border of the file open dialog)

Call DataFileLoadSel("C:\Example.tdm","TDM","[1]/[2,3]","Load")

 will load two channels of the file.

DIAdem will then only load some selected groups/channels

instead of the whole file but the plugin code will not change.

Do you wan't to split up because script is too slow or because you didn't know of selective loading.

[kdaily]

We want to make an On-Demand because the script is slow

[Brad_Turpin]

Hi kdaily,

What is slow exactly, the indexing the DataFinder does with the DataPlugin, or using the DataPlugin to load the zipped binary files into the Data Portal?  Unzipping large binary files will slow things down, regardless of language.  Would it be an option instead to use a smaller binary data type, such as I32 or I16 or even I8 instead of Doubles?  Is there a header file for these binary files?  If so, is it also inside the ZIP file?

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[kdaily]

The DataPlugin to load the binary files is slow. I have not yet made a DataPlugin to open a zip file that contains zipped binary files. First I am trying to understand what needs to be updated using a previous DataPlugin. It is a One-Shot that opens a zip file that contains binary files that are not zipped. When I right click a .zip file that contains the binary files and select Load Data it takes a long time for the data to show in Data Portal and the screen to select files when I right click a .zip file that contains the binary files and select Open Selectively takes a long time to show. My thinking was that I need to make an On-Demand DataPlugin but I don't understand how DIAdem would use it to only open user selected binary files. There is a header file and I don't believe the data type can change.

[Brad_Turpin]

Hi kdaily,

When you talk about on-demand and one-shot DataPlugins, are you referring to DataPlugins written in LabVIEW, as opposed to VBScript or C++?  It's probable that the unzipping is taking a long time because the unzipping routine you're using unzips the entire contents of the zip file, even if you only want a few of the binary files in it.  It's possible that you could change the function you're calling to only unzip the selected binary file(s) you want.

That brings us to the LabVIEW on-demand DataPlugin.  This DataPlugin is called at least once for each channel that is being loaded.  In the case of long data channels, this DataPlugin is called multiple times for a given channel, loading a different block of data values each time.  As such it is critical that an on-demand DataPlugin have random access to both the channels within the data file as well as specific data value ranges within each channel's value array in the data file.  Typically this only really works with binary files.  The hair in the soup for you could be the ZIP file container the binary files are stored in.

Youu could theoretically have the first call to the on-demand DataPlugin unzip the required files to a temporary location, then have subsequent calls use the temporary location, then have the final call delete the temporary files.  I haven't ever tried that, but I'm betting it could work.  The only way to communicated from one call of the on-demand DataPlugin to the next is to write the information you want to transmit to a Channel property and read back that Channel property value the next time the DataPlugin is called (for that Channel).

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[kdaily]

Hi Brad,

Sorry it has taken me so long to respond to your last post. I got roped away on another project. Yes, I was referring to DataPlugins written in LabVIEW. I guess I don’t fully understand how the on-demand DataPlugin works and how to use it to read user selected binary files from the zip file container. All of the examples I have found set up channels and read all data. Is there a Knowledge base article or something that explains how to set up the yourCodeHere_meta.vi and yourCodeHere_raw.vi? Thanks for your help.

[Brad_Turpin]

Hi kdaily,

The "meta" VI is called once at the beginning of the load process.  It is intended to read the metadata from the top of the data file or from a header file, which is required in all cases no matter what channel(s) or value range(s) you are loading.  The "raw" VI is called multiple times.  In the simplest case it is called once per channel, and you load and return the entire value array of that channel each time it is called.  But it is designed to support buffered reads from larger data channels, so your code needs to move the file cursor to the spot in the data file where the start index input is requested the buffer of channel values to start from and read out the number of values requested.  If the start index is 0 and the number of values is the length of the channel, then you will read the whole channel, otherwise you will read only part of the channel each time the "raw" VI is called.  There is documentation in the LabVIEW DataPlugin API that installs along with the VIs.

Does that help?

Brad

[kdaily]

Hi Brad,

I think it helps a little. I’m still trying to figure out what from my problem needs to go into the meta VI and what into the raw VI. I will try your suggestions for a first pass at it.

[kdaily]

Hi Brad,

I did a first pass and the On-Demand DataPlugin seems to be ok. Though I haven’t built/tested the plugin yet, I have only used the test VI in the project. I am unzipping to a temporary location and reading the unzipped binary files from there. The next step is figuring out how to delete the temporary directory after the last iteration of the raw VI runs. Do you have any suggestions?