Memory problems with large 2d text string

[Rob Hingle]

I have an application using a rather large 2d string database of parts in a catalog.  There are columns for part number, description, etc, and about 20,000 entries.  None of the strings are more than 20 characters, and when the data is stored on disk as a saved cluster, the filesize is only 1.6 MB.  Yet when the program runs as an executable, the memory usage almost always exceeds 100MB in Task Manager.  I've been reading about efficient memory usage with arrays but there is little help with string arrays.  If there was a way to allocate the string lengths in the array I suspect this vi would consume less memory.  Another solution mentioned (though this would be an extreme modification) is to have the entire database as a single string with an array of offset and record length pointers to distinguish the individual records.  But I'm not even sure yet if this is a problem, or if its expected for Labview executbales to take up alot of memory.  Does anybody have experience with this sort of program, which involves pulling individual records from the catalog for compiling assembly parts lists, and also searching the entire database for text query matches? 

[DFGray]

Compared to C/C++ executables, LabVIEW executables take up a lot of memory.   However, 100MBytes for the application you mention seems like a bit much.  The actual data should be stored in about the same amount of space in memory as it is on disk.  However, if you don't really pay attention, you can make many copies of that data and end up using far more memory than you need to.  How you use it and store it is important.  Read through Managing Large Data Sets in LabVIEW for pointers on how to minimize those copies and speed performance.  

I would recommend storing each of your data columns in a separate single-element queue.  You can search for anything by dequeueing the array, doing the search on the array, then enqueueing the array again.  This will not produce any copies and will then give you an index to fetch the other data from the other columns.  To fetch data from a column, dequeue it, index the array, and enqueue the array again.  You will make a single copy of the element you fetched.  Since all your single element queues will be of type "queue of array of strings", you can create a couple of subVIs to handle the search and fetch functions.  This also allows you to encapsulate the search and fetch functionality so you can improve it in the future, if necessary (e.g. currently use the LabVIEW search 1D array function but in the future use a hash table or binary search).

Let us know if you need more help.

[Rob Hingle]

Maybe first I should create a simplified version of the code to do comparisons.  One possible additional memory drain is that I am using an ActiveX container web browser to display the search results as a web page with hyperlinks and graphics.  Another little thing is that all the data for the program (large 2d text array plus several numeric arrays and file locations etc) exists in a single cluster that is shift registered around the consumer loop.  I have read though that labview recognizes the unbundle rebundle pattern and does not waste memory when it sees that pattern.  Maybe I should try the manage memory cluster structure?  Another thing is that the arrays are compiled from multiple DBF databases, whose contents are pulled into memory with many more columns than are used.  I index the columns of interest (different column indices for each DBF - thanks previous programmers) and rebuild the array I end up keeping.  Does this keep the full arrays in memory even after the sub vi executes?  Do you have any references on that queue method?

[DFGray]

If you can get to all your data from a local shift register, you won't be able to beat the performance of that with anything.  However, keeping your data as a 2D array of text instead of several 1D arrays, one per column, will cause you searching issues.  If you do unbundle/rebundle correctly, LabVIEW is smart enough to not make copies (the buffer viewer is your friend - use it often).  The inplace structure is a lot more reliable.  You don't have to think as much.  I highly recommend it.

Whether or not LabVIEW keeps the extra data in memory depends on how the program was written.  It may or may not.  You can step through your program with the Windows task manager up and watch for memory allocation issues.  If it does keep the memory, you can then try to eliminate the issues.  There are some VI property flags you can set and primitives you can call to flush excess memory explicitly.  I have never used them, but they were created for cases such as this.

You can find more information on the queue method (and a lot of other methods) in this thread - Reference Objects in LabVIEW - LabVIEW 8.5 Update.

Let me know if you need more information.