Clustering large 2d arrays?

[gstanczak]

I am trying to combine some large 2d arrays (of differing length) into a single indicator, while maintaining the lengths of each.  Building a cluster seems sluggish.  I see comments people have made regarding the fact that this process may be slow, but no solutions.

An alternative that I am trying is recording in a separate array the size of each 2d data array as it is appended to the larger, now 3d, array.  Later, after performing some operations on the 3d array and when I want to use the 2d data for something else, I use the smaller array with the recorded dimensions and the array subset function to trim each back to their original size.

Can anybody comment on this method?  I don't think its consistent with a modular programming approach (something I  am just learning the importance of as a non-programmer), as the array with the dimension sizes has to be transported around with the large data array.  Maybe this is okay, I don't know? Is there a better method of combining arrays, keeping their respective lengths, without using the cluster func.  :0

Thanks a bunch,

Greg

[smercurio_fc]

Were you trying to build a cluster of multiple 2D arrays? I would have gone with an array of clusters, with the cluster containing a 2D array. This way the cluster acts as a "shell", and each element of your 1D array corresponds to each 2D array. Sort of like what's shown in the attached figure. Is this what you saw comments on?

Message Edited by smercurio_fc on 07-17-2009 05:36 PM

[RobCole]

I'm just curious why you are combining the arrays into one indicator. Is this for passing between VIs? If it is just for display, you could use a decoration to make it look like one indicator.

If you are trying to access both arrays in one connector on the connector panel, you will have to take a bit of a processor hit to combine them no matter how you do it.

     Rob

[altenbach]

How different are the 2D sizes and how often do the sizes change?

Is one of the 2D dimensions always the same?

How big are the arrays?

Do you know the final size in the 3rd dimension when the program starts or are you growing to an arbitrary size by appending new slices?

Can you attach a VI with some typical data?

If the data structures are large (and from your description they probably are), you should make sure to operate "in place" as much as you can. Keep things in a shift register and avoid local variables.

[gstanczak]

Ultimately I was trying to build an array of clusters, each cluster having 2 elements (a path and a 2d array of data).  The reason for the cluster was two-fold: 1) to allow for different array sizes, and 2) to easliy find which file the data came from.  I attached an image which hoepfully is helpful.

Thanks again,

Greg

[gstanczak]

Altenbach, I am reading in files with acquired 2d data, channels x samples.

The number of channels is always the same so one dimension is always 16.

The difference in the samples dimension can be anywhere from 5000 to 40000 at the max.

The user choosers how many files to read in (5-10) so the 3rd dimension of the array is known as soon as they choose.

[Darin.K]

My two cents (plus or minus two cents).  If I have a 3d array of floating point data I set the default value of the control to 'NaN'.  This doesn't get in the way of my math operations and it is easy to extract the submatrix later.  There is some hit, since the math operations may be performed on a matrix with a lot of NaNs instead of a smaller matrix.

My second technique is probably a bit dated since I started doing this before LVOOP.  I would use an uninitialized shift-register in a while loop inside a sub-vi to store the data.  The shift register contained a 3D array.  Inside the while loop is a case structure wired to an enumerated type used to select the method (constructor, getter, setter, etc.).  The method to add a 2D array would update a second array to track the sizes of the individual matrices.  The method to retrieve one of the matrices would use the size array to only return the proper subset.  The downside is that you have to write all of the methods (add a matrix, remove, replace,...), but it seemed the most efficient way to move around large datasets, even when they are all the same size.  I'd share some code but I can't open my version 3 files (perhaps a new thread). 

These were truly large datasets in the days of measuring RAM in Mb, so I would explore the cluster option first.

-Darin

[smercurio_fc]

You appear to be using an old version of LabVIEW. I say this based on the icon for the VI that's reading the file. That appears to be the old Read Characters From File from LabVIEW 7. What version of LabVIEW are you using? This has an impact.

[gstanczak]

Smercurio_fc, I am using 7.1.1