Exclude DataFinder Search Area by File Date?

[Julia Dawkins]

Hello, I am using DFSE 2.0 indexing some folders with ~many hundreds GB of data (index size ~12 GB). 

As our data continues to grow, I think it would be neat to limit my indexed files by date.  I don't want to delete old data, but I would prefer to not keep them in my index.  Unfortunatly, we do not structure our file storage in a way where I could 'exclude' folders (each folder is a mix of new/old data). 

Is there a way I could accomplish my data management dream?  I admit I have not read any help files, etc.....

Thanks!

Julia

[Brad_Turpin]

Hi Julia,

Sort of.  My recomendation would be for you to write a "post-processing" DataPlugin for each base DataPlugin (TDM, TDMS, CSV) that you use in your DataFinder.  The post-processing DataPlugin could check the File.DateTime property or the created date of the file and throw an error if the date is out of range.  This would have the advantage that your searches would return only files with valid dates, and your data base would be much smaller than it currently is, because the invalid date range files would only add a File-level "could not index" record to the data base, rather than adding all the Groups and Channels and their properties, as is currently happening.

This is of course not exactly what you've asked for, but it is the closest thing I can think of for the DIAdem 11.0 time frame.  For the case of DataPlugins you wrote yourself, you could build in the datetime check into the primary DataPlugin and avoid the need for a post-processing DataPlugin.  Ask if you have questions about what a post-processing DataPlugin is and how to program one.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Julia Dawkins]

Thanks Brad!  That sounds like a great work-around.  Do you have an example of a post-processing DataPlugin that I could look at? 

[Brad_Turpin]

Hi Julia,

Actually, the example DataPlugin that you get when you create a new post-processing DataPlugin would work for you.  It's general and uses recursive subroutine calls in order to traverse N hierarchy levels (possibly much more than 3), but it runs just fine with TDM files.  My guess is that you only have DataPlugins with exactly 3 hierarchy levels, so I created a simpler post-processing DataPlugin that assumes this explicitly.  I also added in the freshness date test, assuming that you wanted to use the "Storage DateTime" File property for comparison purposes.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Brad_Turpin]

Hi Julia,

This version of hte DataPlugin is a little simpler-- it assumes only 1 Root, which I assume will also always be true for your DataPlugins.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Julia Dawkins]

Hi Brad,

I'm not sure if I am implementing this correctly.  I created a new DataFinder for testing, and was hoping to be able to exclude all dataplugins except for the new TDM Recent...but I am unable to de-select the TDM or TDMS option.  I am not sure how this will work .. won't the old files get indexed with the regular TDM dataplugin? 

Julia

[Brad_Turpin]

Hi Julia,

You can not disable the TDM or TDMS DataPlugin in the DataFinderr configuration, but you can define the post-processing DataPlugin to be of higher priority in the "File Extensions" tab of the DataFinder configuration dialog.  This way the freshness-date-checking TDM DataPlugin will run first.  But you're right, I had forgotten that the regular TDM DataPlugin will pick up as soon as the freshness-date-checking TDM DataPlugin throws an error.  So in that DataPlugin instead of calling RaiseError if the freshness date is wrong, you'll need to declare a dummy Group, say named "Old", with no channels assigned to it.  Here's a modified version that does that.

Brad Turpin
DIAdem Product Support Engineer
National Instruments