Data not being read with custom Data Plugin

[Shahrukh]

Hello,

I'm trying to setup an on-demand binary data plugin to read our custom data format.  Following the plugin SDK instructions, I wrote code to read the header and data from our format, and it works using the testing/debugging VIs included withn the SDK.  

Unfortunately, when I compile and install the plugin, and try to access my data in Diadem, while the individual channels are successfully created, they don't contain any the data. 

I've attached a zip file with all the relevant files, would appreciate it if someone could take a look and see if they can find anything wrong.

Thank you,

Shahrukh Alavi

National Research Council Canada

[Mavis]

Hi Shahrukh,

I gave it a simple test and found this could be the root cause.

In PB_Header.vi, the subVI Create On Demand Channel.vi doesn't have anything wired to its Channel Length input. If not wired, the default value is 0.

The Channel Length information is very important to an on-demand DataPlugin, since it tells its hoster whether/how to call read_raw_data.vi.

Please wire a correct value to this input and build DLL/installer again.

Hope this helps,

Mavis

[Shahrukh]

Hi Mavis.
 
I have wired my waveform length into the channel length input of the Create On Demand Channel subvi.  When trying in diadem, it now seems to read the data (the status bar shows some activity) but once its complete, I get the following error:  "Cannot import all channels or channel data from this data store".  Now again, the channels are listed in the data portal, but no data is present.

I also found that I wasn't handling the Count=-1 input properly in my PB_Data.vi, which I've now corrected.  New files attached.

One question I had was how diadem uses the on-demand plugins.  When I drag a file from the Navigator screen to the Data Portal, does it only read only the channel header information, then load the data when I select a parameter from the list or load all the data at once? 

Appreciate the help.

Shahrukh

Message Edited by Shahrukh on 01-27-2010 09:23 AM

[Brad_Turpin]

Hi Shahrukh,

I'm hoping that Mavis will continue to assist you with debugging your DataPlugin, but I'd like to answer your question about how DIAdem will use an on-demand LabVIEW DataPlugin.  The short answer is that it will handle and access the data files you wrote your LabVIEW program to read just like DIAdem handles its own native TDM and TDMS file formats, which is to say differently depending on the current DIAdem acitivity:

1)  When the DataFinder is indexing the data file, it only looks at the header information and NONE of the data values into the Data Portal

2)  When you drag the data file from the NAVIGATOR to the Data Portal, DIAdem will look at the header and load ALL the data values

3)  When you right-click on the file and use the "Data Reduction..." menu, DIAdem will look at the header and load only the data values (array index range) that you specify into the Data Portal.

4)  When you right-click on the file and use the "Register Data" menu, DIAdem will look at the header only and create dynamic, on-demand data channels in the Data Portal which appear greyed-out.  Any time thereafter that you graph these "registered" channels or run ANALYSIS routines on them, DIAdem will automatically go back to the data file's location and load the channel values buffer by buffer directly from the data file-- using the data file as in-place, DIAdem-managed virtual memory.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Mavis · Accepted solution]

Hi Shahrukh,

The status bar and the error message often suggest a failure in loading last parts of channel values.

We can use Create DataPlugin Output File.vi to debug our code.

The data file appears to have 7 channels, each has 40282 data points. So I set Offset to 40280 and # Points per buffer to 2 and got error 4 " End of file encountered." in PB_Data.vi. Could it be one of the following reasons ?

1. Number of data points (waveform length) is not correct.

2. The file offset calculation is not correct.

Mavis

[Shahrukh]

Brad:  Thanks for that clarification, helps me understand the plugins better.  As a second question, can you explain how this "offset" paramater is used, if I'm loading all the data for a channel, does it read it all with one command of offset=0, count=max or does it in smaller groups like 1000 at a time.

Mavis:  I was also thinking that it was some boundary condition that was giving the error.  I'm pretty sure the data size is correct, which leaves the offset function.  I'm still working with the offset functionality, because I never had it working.  I get an error in the "set properties" anytime the offset (index) isn't "0".  The same thing happens in the other plugin examples.  Is this because the channel is currently empty, therefore the specified index doesn't exist?Another thing I noticed was I get a warning "2552" in the write data(channel group) call, when I am making my channel groups. 

I'll spend some more time on it in the next few days.

Thanks,

Shahrukh

[Mavis]

Hi Shahrukh,

---

I get an error in the "set properties" anytime the offset (index) isn't "0".  The same thing happens in the other plugin examples.  Is this because the channel is currently empty, therefore the specified index doesn't exist?

---

This is exactly the reason.

The warning "2552" is expected when new object/property being created.

The "offset" question: a full channel could be loaded in small groups for optimization, thus we need the offset parameter.

Mavis

[Brad_Turpin]

Hi Shahrukh,

An on-demand DataPlugin gives you the ability to request subset array buffers from one or more data channels, but if you ask the DataPlugin to load all the values, it will return one really BIG buffer of values for each data channel.  The software layer that DataPlugins "plu into", called USI (Universal Storage Interface), always calls a DataPlugin once for each data channel that needs to be loaded.  So even if you ask for all the values of all the channels, the DataPlugin be called first to load all the values from the first channel, then again to load all the values from the second channel, etc.

Now DIAdem is clever enough to always ask for the values of any huge channel in multiple buffers, so just dragging a file from the NAVIGATOR into the Data Portal kicks of a process that loads optimized buffers from each data channel the DataPlugin loads.  If you're using a DataPlugin in LabVIEW, you need to know what you're doing.  Of course in the case of a G-code DataPlugin, you should just use the source code of the DataPlugin directly if you want to load data into LabVIEW-- it'll be more efficient, and you'll have more control.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Shahrukh]

Update:  I found the faulty logic in my code, where I was reading x-1 points, thus the errors in Diadem.  Everything seems to be working well now.

Thanks for you help.