How to handle massive array operations

[Gurdas]

Hi!,

We are working with large arrays of the order of 5000 x 100000 (rows x col).
We wish to perform typical array operations like replace subset, indexing, rotating and addition.

Our questions:

1) How do we work with this array within the same VI without creating indicators or controls. Wiring, we understand, is an option but then the wires will clutter our screen.

2) How do we pass/retrieve this array data to/from sub-VIs without creating indicators or controls.

3) What the fastest recommended methods of handling such large arrays.

4) Our array will be either Boolean or U8. Will both data types give similar speed results?

5) When using the replace subset function, can the subset be same dimension as original array?

6) Can we rotate a particular row within a 2D array? As of now, we extract the row as 1D, rotate and then replace the original row with rotated row.

Any ideas/suggestions (specific to array handling) to help speed our VI will be highly appreciated.

We are using LV 7.1 on Win2000

Thanks,
Gurdas

[Benoit]

If the control and the indicator is in a subVI, the performance of LabView will not be degrated by the speed of screen refresh.

Boolean operation is faster, but keep in mind that one bit take 8 bits in memory.

When you use the replace array subset, the array as the same size. Try with a smaller VI to test the functionality of your app.

For your question 6, you got it....

Benoit

[Gurdas]

Thanks Benoit.

I am not clear about "When you use the replace array subset, the array as the same size."

Do you mean to say that if my original array is 2D, I can have a subset array which is also 2D and replace it in a single shot? I have my doubts on this? How will I tell the replace function where my subset whould go?

For example, say my original array is 25 x 40 (rows x cols). I have another array of size 6 x 8 which I would like to replace into my original such that it overwrites rows 12 to 17 (6 rows) and columns 21 to 28 (8 cols).

Rgds,
Gurdas

[Benoit]

If you want to replace with a 2D array, you will need to make it with a for loop.

Benoit

[Underflow]

Gurdas,

Check out
Managing Large Data Sets in LabVIEW and the links it contains.

In particular, it will help with your item #2, and
give you a better overview of the subject.

Joe Z.

[smercurio_fc]

Actually, that's not true. If you want to replace a 2D array subset of a 2D array the Replace Array Subset will allow you to wire in a 2D array. You just need to specify the row and col indices where you want the replacement to occur. See attached VI as an example.

As for your other questions:

(1) Not sure what you mean by this. If you need to work with that data in the same VI you use wires - that's the most efficient way of doing it. Are you thinking about local variables or something?
(2) Controls/indicators are the best way to pass data in/out of VIs. LabVIEW only makes a copy of the data if the subVI front panel is open in order to display it on the front panel.
(3) Let LabVIEW handle the memory management as this is done automatically.
(4) LabVIEW stores Booleans as 8-bit data. You should choose the one that makes more sense for your application.
(5) Yes, see previous response.
(6) You will need to be a little more specifc about this one, though I suspect the Transpose 2D Array function coupled with the Replace Array Subset is what you're looking for.

-Saverio

[jasonhill]

How will I tell the replace function where my subset whould go?

When you plug your array into the input of the Replace Subset, it should automatically resize to give two index inputs.  One is for the start row, the other is the start column.  Wire both of these to control where the 2D subarray is inserted.

You can also resize the Replace Subset to perform two separate actions on two separate parts of the array.  (see attached)  I cannot say offhand if this will help with your memory management.  You will have to do a bit of profiling to find the most efficient way to handle this.

I work with aproximately a tenth the data you are looking at and constantly run into out of memory issues.  The link Underflow mentioned is extremely valuable.

[Gurdas]

Saverio, Joe, Jason and Benoit - Thanks!

Reply to Saverio's inputs:

(1) Not sure what you mean by this. If you need to work with that data in the same VI you use wires - that's the most efficient way of doing it. Are you thinking about local variables or something?
No, we avoid locals like birdflu! What I meant was how do I carry data from one part of my block diagram (BD) to another without using wires (because wires clutter the block diagram)? Is there no variable/container which will hold data without it being a control/indicator? Some of my front panels will be displayed. In such VIs, the moment I use a control/indicator in the BD to "hold" my data, my speeds drop.

(2) OK

(3) OK

(4) LabVIEW stores Booleans as 8-bit data. You should choose the one that makes more sense for your application.
U16 makes most sense.
U8 also does the task, only catch is at the end I need to convert the array into U16 for the last operation to happen (all elements in a col are added; thus final array is 1D row array).
I would use Boolean ONLY if it gives significant speed gains over U8 (I will need to convert Boolean to U16 at the end).
Any inputs/ideas?

(5) OK

(6) You will need to be a little more specifc about this one, though I suspect the Transpose 2D Array function coupled with the Replace Array Subset is what you're looking for.
Lets say the first row of my 2D array has 5 elements, which are 3 6 12 8 1. I want this to become 12 8 1 3 6. In effect I left rotated my first row my two positions.

Rgds,

Gurdas

[smercurio_fc]

Gurdas,

With respect to your responses:

(1) If you don't want to use wires then local variables, global variables, or shared variables (LabVIEW 😎 are your only other options. Any of these (especially the global variables) are going to give you a big impact on speed. Stick with wires to pass data around on the block diagram.

(4) Stick with U16 then. At the point where you intend to do the conversion to U16 LabVIEW is going to need to allocate a new memory buffer to do the conversion, and if your array is large this is going to translate into a big performance hit. With very large arrays you want to minimize or eliminate data type conversions.

(6) What you would need is to peel off those elements using an Index Array function and then use the Rotate 1D array function. See attached updated example.

[jasonhill]

To rotate a single row without excessive data duplication, you can delete the row in question, rotate it, and reinsert it (see attached).  I am not certain if Labview attempts to resize the array with each delete and insertion.  Array resizing may have a performance hit, but I do not think it would be as large as carrying around an extra copy of the entire array.  Again, I reccomend profiling a couple different methods and using the one that works best.

With regard to "wire clutter"... IMHO astetics and other programming practices will ocassionally have to take a back seat to performance.  Good documentation can mitigate a lot of the confusion associated with railyard-like block diagrams.