topic Re: How to handle massive array operations in LabVIEW

How to handle massive array operations

Gurdas — Mon, 30 Jan 2006 17:05:18 GMT

Hi!,

We are working with large arrays of the order of 5000 x 100000 (rows x col).
We wish to perform typical array operations like replace subset, indexing, rotating and addition.

Our questions:

1) How do we work with this array within the same VI without creating indicators or controls. Wiring, we understand, is an option but then the wires will clutter our screen.

2) How do we pass/retrieve this array data to/from sub-VIs without creating indicators or controls.

3) What the fastest recommended methods of handling such large arrays.

4) Our array will be either Boolean or U8. Will both data types give similar speed results?

5) When using the replace subset function, can the subset be same dimension as original array?

6) Can we rotate a particular row within a 2D array? As of now, we extract the row as 1D, rotate and then replace the original row with rotated row.

Any ideas/suggestions (specific to array handling) to help speed our VI will be highly appreciated.

We are using LV 7.1 on Win2000

Thanks,
Gurdas

Re: How to handle massive array operations

Benoit — Mon, 30 Jan 2006 18:04:58 GMT

If the control and the indicator is in a subVI, the performance of LabView will not be degrated by the speed of screen refresh.

Boolean operation is faster, but keep in mind that one bit take 8 bits in memory.

When you use the replace array subset, the array as the same size. Try with a smaller VI to test the functionality of your app.

For your question 6, you got it....

Benoit

Re: How to handle massive array operations

Gurdas — Mon, 30 Jan 2006 18:13:45 GMT

Thanks Benoit.

I am not clear about "When you use the replace array subset, the array as the same size."

Do you mean to say that if my original array is 2D, I can have a subset array which is also 2D and replace it in a single shot? I have my doubts on this? How will I tell the replace function where my subset whould go?

For example, say my original array is 25 x 40 (rows x cols). I have another array of size 6 x 8 which I would like to replace into my original such that it overwrites rows 12 to 17 (6 rows) and columns 21 to 28 (8 cols).

Rgds,
Gurdas

Re: How to handle massive array operations

Benoit — Mon, 30 Jan 2006 18:23:49 GMT

If you want to replace with a 2D array, you will need to make it with a for loop.

Benoit

Re: How to handle massive array operations

Underflow — Mon, 30 Jan 2006 19:20:02 GMT

Gurdas,

Check out
Managing Large Data Sets in LabVIEW and the links it contains.

In particular, it will help with your item #2, and
give you a better overview of the subject.

Joe Z.

Re: How to handle massive array operations

smercurio_fc — Mon, 30 Jan 2006 19:20:08 GMT

Actually, that's not true. If you want to replace a 2D array subset of a 2D array the Replace Array Subset will allow you to wire in a 2D array. You just need to specify the row and col indices where you want the replacement to occur. See attached VI as an example.

As for your other questions:

(1) Not sure what you mean by this. If you need to work with that data in the same VI you use wires - that's the most efficient way of doing it. Are you thinking about local variables or something?
(2) Controls/indicators are the best way to pass data in/out of VIs. LabVIEW only makes a copy of the data if the subVI front panel is open in order to display it on the front panel.
(3) Let LabVIEW handle the memory management as this is done automatically.
(4) LabVIEW stores Booleans as 8-bit data. You should choose the one that makes more sense for your application.
(5) Yes, see previous response.
(6) You will need to be a little more specifc about this one, though I suspect the Transpose 2D Array function coupled with the Replace Array Subset is what you're looking for.

-Saverio

Re: How to handle massive array operations

jasonhill — Mon, 30 Jan 2006 19:26:27 GMT

How will I tell the replace function where my subset whould go?

When you plug your array into the input of the Replace Subset, it should automatically resize to give two index inputs. One is for the start row, the other is the start column. Wire both of these to control where the 2D subarray is inserted.

You can also resize the Replace Subset to perform two separate actions on two separate parts of the array. (see attached) I cannot say offhand if this will help with your memory management. You will have to do a bit of profiling to find the most efficient way to handle this.

I work with aproximately a tenth the data you are looking at and constantly run into out of memory issues. The link Underflow mentioned is extremely valuable.

Re: How to handle massive array operations

Gurdas — Tue, 31 Jan 2006 05:25:20 GMT

Saverio, Joe, Jason and Benoit - Thanks!

Reply to Saverio's inputs:

(1) Not sure what you mean by this. If you need to work with that data in the same VI you use wires - that's the most efficient way of doing it. Are you thinking about local variables or something?
No, we avoid locals like birdflu! What I meant was how do I carry data from one part of my block diagram (BD) to another without using wires (because wires clutter the block diagram)? Is there no variable/container which will hold data without it being a control/indicator? Some of my front panels will be displayed. In such VIs, the moment I use a control/indicator in the BD to "hold" my data, my speeds drop.

(2) OK

(3) OK

(4) LabVIEW stores Booleans as 8-bit data. You should choose the one that makes more sense for your application.
U16 makes most sense.
U8 also does the task, only catch is at the end I need to convert the array into U16 for the last operation to happen (all elements in a col are added; thus final array is 1D row array).
I would use Boolean ONLY if it gives significant speed gains over U8 (I will need to convert Boolean to U16 at the end).
Any inputs/ideas?

(5) OK

(6) You will need to be a little more specifc about this one, though I suspect the Transpose 2D Array function coupled with the Replace Array Subset is what you're looking for.
Lets say the first row of my 2D array has 5 elements, which are 3 6 12 8 1. I want this to become 12 8 1 3 6. In effect I left rotated my first row my two positions.

Rgds,

Gurdas

Re: How to handle massive array operations

smercurio_fc — Tue, 31 Jan 2006 15:08:54 GMT

Gurdas,

With respect to your responses:

(1) If you don't want to use wires then local variables, global variables, or shared variables (LabVIEW 😎 are your only other options. Any of these (especially the global variables) are going to give you a big impact on speed. Stick with wires to pass data around on the block diagram.

(4) Stick with U16 then. At the point where you intend to do the conversion to U16 LabVIEW is going to need to allocate a new memory buffer to do the conversion, and if your array is large this is going to translate into a big performance hit. With very large arrays you want to minimize or eliminate data type conversions.

(6) What you would need is to peel off those elements using an Index Array function and then use the Rotate 1D array function. See attached updated example.

Re: How to handle massive array operations

jasonhill — Tue, 31 Jan 2006 15:33:21 GMT

To rotate a single row without excessive data duplication, you can delete the row in question, rotate it, and reinsert it (see attached). I am not certain if Labview attempts to resize the array with each delete and insertion. Array resizing may have a performance hit, but I do not think it would be as large as carrying around an extra copy of the entire array. Again, I reccomend profiling a couple different methods and using the one that works best.

With regard to "wire clutter"... IMHO astetics and other programming practices will ocassionally have to take a back seat to performance. Good documentation can mitigate a lot of the confusion associated with railyard-like block diagrams.

Re: How to handle massive array operations

Gurdas — Tue, 31 Jan 2006 16:35:38 GMT

Jason,

Why delete, rotate and then reinsert?

Why not just extract the row, rotate and replace? I thought this might be faster.

- Gurdas

Re: How to handle massive array operations

Gurdas — Tue, 31 Jan 2006 16:48:48 GMT

Saverio,

That clarifies.

All,

We are currently taking upward of 60 secs to process our array. The target is less than 1 sec!

We have hit upon a few ideas which involve 'tricks' specific to our application. Shall post results/insights as they become available.

I still need inputs on what is faster: replace row in an array OR insert row into array? Further, if the insertion is always at end of array, is it not same as building an array (which I know to be bad at speed because of periodic reallocation)?

Thanks,

Gurdas

Re: How to handle massive array operations

Darren — Tue, 31 Jan 2006 16:55:42 GMT

Hi Gurdas,

In general, replacing array elements will always be more memory efficient than inserting array elements. This is because replacing array elements does not involve a memory reallocation, while increasing the size of an array does.

Of course, this rule goes out the window if your array elements are not flat datatypes. All the memory reallocation advice everybody is giving assumes your arrays are numerics or booleans.

-D

Re: How to handle massive array operations

smercurio_fc — Tue, 31 Jan 2006 17:05:16 GMT

60 seconds down to 1 second. Well, that can be a bit of a challenge.

If you post what you're doing with the array perhaps someone can offer suggestions to improve the speed...

Re: How to handle massive array operations

Ben — Tue, 31 Jan 2006 17:25:59 GMT

"If you post what you're doing with the array perhaps someone can offer suggestions to improve the speed..."

Yes, please do!

Curious,

Ben

Re: How to handle massive array operations

Gurdas — Tue, 31 Jan 2006 18:54:20 GMT

Sorry folks, cannot post the code (or even sections).

If the questions start becoming tricky and it will be help to have code accompany them, I shall build dummy VIs for the forums. But, lets hope I don't need to do that because I am really short of time right now!

- Gurdas

Re: How to handle massive array operations

tst — Tue, 31 Jan 2006 20:30:11 GMT

@Gurdas wrote:

2) How do we pass/retrieve this array data to/from sub-VIs without creating indicators or controls.

I haven't read the posts in this thread, but you might want to have a look at JPD's attachment here for an example of a global array which keeps the memory consumption down.

Re: How to handle massive array operations

Gurdas — Tue, 14 Feb 2006 12:39:04 GMT

All,

We managed to beat the problem and our own expectations by a huge margin. And it feels great 🙂

While there was some scope for improvement in our use of LV syntax, we soon realised that it might return a maximum benefit of only a few seconds (say 5 secs; while the target was from 60s to 1s!). So, our problem forced us to think out-of-the-box and we relooked at the application from scratch. Some 3 days of brainstorming later, we now have the array worked, reshaped and cooked in less than 5ms!! This is such an important milestone that we stand in awe at the sheer power of an idea. More so because the array operation (which was taking 60s) has to be done atleast a few thousand times to get a final optimised array. With speeds of 5ms, we can now confidently run the optimiser for long enough to guarantee a global optima.

We learnt a few lessons, which I would like to share with the LV community:

1) For most of the time (shall I say 95%), if you strictly adhere to the basic thumbrules of good programming, you should be fine. The esoteric features of any language should be left for esoteric functions. Our LV team might get just 4/10 on LV skills but we still manage to write some very good and robust software. I feel greedy to write down my list of top10 rules to follow but I'll leave that for another day and thread.

2) More than programming skills, focus on domain knowledge and a true understanding of the problem. The advantages from good coding are just a fraction of the advantages that come from good algorithms.

3) Use "engineering" tools (like VI profiler) to trap the larger pitfalls in your code. You can achieve 80% improvement in application speed and robustness by just focussing on the largest 20% pitfalls.

4) When the going gets tough, send a post to discussion forums, sip a cup of piping hot coffee and sleep over the problem for a night 🙂

To know a little more about the application (SilentRoll), you could visit http://www.qagetech.com/SilentRoll.html

Rgds,

Gurdas

Re: How to handle massive array operations

tst — Tue, 14 Feb 2006 13:58:22 GMT

What did you actually do to speed up the operation?

Was it anything clever that we would like to know about or was it just cleaning the program to improve the algorithm and avoid things like multiple copies?

Re: How to handle massive array operations

Joe_H — Tue, 14 Feb 2006 14:11:37 GMT

I've been reading this thread and would love to know the tricks you did to pull it off. i think it would be beneficial to the LV community to know the way you really solved the problem.