Exclude directories from "Recursive File List.vi"

[SeanDonner]

Hi all,

I was hoping for some best practices on how to exclude certain directories from the "Recursive File List.vi", specficially the '.svn' folder in my case (although their may be more folders to exclude in the future).  Seems to me that there is no way to get around some post-processing of the path array output.  Since the path array is already pre-sorted in a sense, my idea is to somehow find the first and last index of where '.svn' show up, then use this information to send to 'Delete From Array'.  "Search 1D Array" will give me the index of the first entry, but not the last.  Perhaps I can do a seperate "Recursive File List" on just the .svn directory and use the "Number of Files" output as the length input to "Delete From Array" ?

Thanks for your help and suggestions.

[Darren]

Hi Siege,

With what you currently have available, post-processing of the path array is probably the best solution.  In an ideal world, Recursive File List.vi would have a way to exclude folders from the results.  I think this is a good idea, and since we have extra terminals available on that VI, I could probably make it happen in a future LabVIEW release.

Would you want it to be a string array, where you could specify an array of folder names?  Any folder encountered during the recursion that matches any folder name in the array would not be included in the results.  But wait a sec...is the folder name you want to exclude simply called ".svn", or is it [somename].svn?  If so, then maybe we'd need the exclusions array to be an array of patterns (like *.svn) instead of an array of names?  Let me know if this is the implementation you were thinking of.

In the meantime, if you don't like the idea of post-processing, you could always make your own copy of Recursive File List.vi and have it exclude certain folders from the recursion.

-D

[waldemar.hersacher]

Hi Darren.

folders with the name .svn or _svn ( you can set in the properties of Subversion either to use .svn or _svn) are containing managment information for Subversion. A .svn folder will contain several subfolders. Moving these folders arround machines will corrupt the information for Subversion when a file was updated (or  checked out as MS says) from the repository.

For the request of Siege just a list of foldernames would be valid. for a more general solution a list of foldernames with wildcards or a regular expression should be implemented.

Since we are handling with foldernames I prefer to have an array of paths instead of strings. In the context of files I only use strings to separate the extentions from the base name.

[SeanDonner]

Hi Darren,

Thanks for replying, glad to see you think this option might be nice to have in upcomming versions.  Until you mentioned it, It totally escaped my mind that "Recursive File List" was not a LabVIEW primitive and could be edited.  I decided to take on my feature request as a learning experience for my CLD.  After some thought, it actually wasn't that hard, just a case statement around the majority of the code to skip a folder if it matched the pattern.  I had to do a minor tweak with the way "All Folders" was being created, I switched it from an auto-index to a build-array through a shift register so that an excluded directory would not show up in the list.  

Also, as a bonus project I decided that since the "pattern" control was using globs for its pattern matching, it might be fun to create a glob --> regex conversion VI so that my new "exclude list" control can also use globs.  Since I use "Match Pattern", I thought it might be confusing if one control was using regex and the other was using globs.  I couldn't find any builtin VI that matched a string on a glob so I think this conversion VI was necessary.  Is the "List Folder" VI the only one left that actually uses globs?

 

Attached are the two VI's that I created, one for the Recursive File List with exclusion support and one for the Glob-to-Regex.  Feel free to use any of this code with no strings attached.

Message Edited by SiegeX on 01-28-2009 05:24 AM

[SeanDonner]

Looking through the VI's some more, I came across an interesting bug.  The "Match Pattern" and "Search and Replace Pattern" VI's have icons which are misleading.  For example, if you were to use the input string "abbc" and pattern string "b*" for the "Match Pattern" VI, the resulting match will be the empty string, not "bb" as the icon suggests.  This is correct behavor as b* will match 'b' zero times, and since the string starts off with a non-b character, this is a match. To get the behavior which the icon suggests, you would use the pattern string "b+" so that it must match at least one 'b' character.

Also, im curious why there exists the "Search and Replace Pattern" VI when you have the LabVIEW primitive "Search and Replace String" which has a Regular Expression Mode.  Is this just for backwards compatibility?

Message Edited by SiegeX on 01-28-2009 04:17 PM

[Darren]

Hi Siege and Waldemar,

Wow, lots of stuff to respond to on this thread.  Here goes:

Waldemar:  The reason I suggested folder name matching instead of path matching was simply because I assume you would want a VI you wrote to recurse folders to work regardless of the absolute paths of the folders you're working on.  For example, if I wanted to exclude ".svn" folders from a recursion for an app I was working on, and my app was in c:\MyApp, couldn't those ".svn" folders reside in any subfolder of c:\MyApp?  And what about if I gave my VI to you, and your app was in c:\source\MyApp?  Any hard-coded paths to ".svn" folders I wanted to exclude would not be excluded because the absolute path was different.  So I'm thinking we'll want to go with folder names here.

Siege(1):  I have never heard the term "glob" used before in this context.  I assume it's some kind of specific pattern associated with files, but according to the LabVIEW Help, the List Folder function only recognizes the * and ? wildcards.  I was planning on doing my implementation where the patterns for exclusion would use the same expression types as the Match Pattern function...obviously we would document this.

Siege(2):  Related to that comment, the reason we still have the Search and Replace Pattern VI is because it uses the Match Pattern function under the hood.  So Match Pattern/Search and Replace Pattern use one matching technique, and Match Regular Expression/Search and Replace String use a different matching technique.  I'll be honest, I still use Match Pattern/Search and Replace Pattern because (a) they're faster than the regular expression stuff and (b) I'm comfortable enough with the patterns used by Match Pattern, and I haven't had a need for any of the more complex patterns that Match Regular Expression provides.

Siege(3):  For my implementation, I was planning on stripping the folder name off the path and doing the match against that, as opposed to doing a match on the whole path.

Siege(4):  I can't believe I never noticed that mistake in the icons of Match Pattern/Search and Replace Pattern before!  I have filed CAR# 143226 against the Match Pattern function and CAR# 143228 against the Search and Replace Pattern VI to fix their icons.

So to summarize, if/when I get to add this functionality to the shipping VI, I'm planning on providing an array of excluded folder name patterns.  If any folder we encounter during the recursion matches the pattern of any of the excluded folder name patterns (based on the matching scheme of the Match Pattern function), we will not recurse its contents, and we will not include it in the list of recursed folders output by the VI.  Does that sound like a good solution?  Oh, and I'll probably add a check at the beginning that sees if the exclusion array is empty...if so, I'll skip the extra logic for the exclusion stuff and just run exactly the same code you see in the VI right now, since it's more efficient, and will probably be the 99% use case.

-D

[waldemar.hersacher]

Hi Darren,

I think we have a little misunderstanding here.

You wrote:

Would you want it to be a string array

I wrote:

Since we are handling with foldernames I prefer to have an array of paths instead of strings.

I wanted to have the data type changed from STR to PATH. The matching itself is on basis of the folder names of course.

[Darren]

Hi Waldemar and Siege,

If the exclusion list was an array of paths instead of an array of strings, we wouldn't be able to specify patterns.  Now that I think about it, maybe pattern matching isn't such a good idea.  For example, if you only wanted to remove folders called ".svn", and you wanted it to be case-insensitive (on Windows), then the pattern you would specify would be "^\.[Ss][Vv][Nn]$".  I don't think that would be very obvious to the average user.  🙂  So maybe we'll just make it simple exact name matching, and it will be case-insensitive on Windows and Mac?  I think we should have it be case-sensitive on Linux since you can have subfolders of different casing (like 'abc' and 'Abc') in the same folder.

I still maintain that it should be an array of strings, though, since we're matching the name of the folder for the exclusion.  If it were an array of paths, I'd still have to convert each path to a string anyway to compare the folder names.

-D

 P.S. - Siege, I noticed you mentioned in an earlier post you're planning to take the CLD.  Make sure to read my blog post on how I prepared for the CLD.

[SeanDonner]

Hi Darren,

Regarding 'globs',  this is a Unix/Linux term for pattern matching which is often used for shell programming.  I think wikipedia has the best definition:

---

glob() is a Unix library function that expands file names using a pattern-matching notation reminiscent of regular expression syntax but without the expressive power of true regular expressions.

---

You brought up possible confusion about case sensitivity (and i'll address that in a bit), but what's even more confusing would be mixing a VI that uses both globular syntax (pattern control) and regular expression syntax (exclude list control).  The confusion will arrise from the fact that the asterisk '*' and period '.' have very different meanings between globs and regex.  A glob pattern of "foo.*" should match only "foo.jpg", "foo.gif", "foo.txt" etc.  Where as "foo.*" as a regex will match 'foo' followed by *anything*.  If you wanted a regex to match the equivilent of the glob foo.*, then you would need the pattern "^foo\..*$", which is exactly what my Glob -> Regex VI does.  It basically escapes out all regex special characters to make them literal and then transliterates '*' to '.*' and '?' to '.' and encapsulates the whole pattern in ^ $ anchors.  Documentation to explain this difference would certainly help, but I feel consistent use of pattern matching syntax would be better.

Now in regards to case sensitivity.  My approach above was using the "Match Pattern" VI to detect whether or not the folder should be excluded, but this does not offer us case insensitive matching.  However, "Search and Replace String" *does* have this ability.  Since we are going to exclude the folder anyway, it doesn't matter that we are replacing its name with the empty string, and we can use the "# of matches" output as a signal to the case structure for when to exclude this folder.  I aslo had to flip the "do nothing" case to TRUE to keep the logic correct.

Also, I took your advice and used the "Strip Path" VI to only match the folder name rather than the whole path as I agree that this will reflect more of what the user is expecting.  However, regarding the case statement to determine whether or not to do any of this if the 'exclude list' control is empty, correct me if I'm wrong but I don't think this is necessary as I am using the for-loop in auto-deindexing mode which should iterate 0 times if the array is empty, esentially doing the check for us.

Message Edited by SiegeX on 01-30-2009 04:26 PM

[SeanDonner]

Here is a picture of the relevant changes to "Recursive File List_Exclude.vi"

 

Below is a test VI that I made to show that case insensitive matching works as well as having multiple patterns to exclude.

 

P.S.  Thanks for the link to the CLD blog,  I can us all the advice I can get.  Also, that was some pretty fancy footwork you did at NIWeek last year on the speed-coding contest.  The usefullness of your Quick Drop feature really shined!

Message Edited by SiegeX on 01-30-2009 04:18 PM