how do i create arrays more efficiently

[eb dub]

how do i create arrays more efficiently?... need to insert data into an array every loop iteration.  I am using the insert into array function and shift registers.  I have been told this is inefficient... and they are right because I can make the program stop and it gives me the error that labview does not have enough memory.  A coworker said to avoid shift registers and investigate property nodes... i am new at this and don't know what this is exactly.. is there a tutorial on efficent data handling.. or examples that would be usefull? 

[GregFreeman]

---

eb dub wrote:
A coworker said to avoid shift registers and investigate property nodes

---

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO! That's almost as good as the code I had that said "Don't use shift registers with large array if at all possible. LV makes a copy of the contents of a shift register for each loop iteration". Not true, shift registers are just a reference to a memory location.

There is a tutorial on managing large datasets, i'll try to dig it up. Do a search of the forums though and you should find stuff. We had an example at an NI development day that showed how you can use named queues to represent an array, then dequeue and enqueue elements

That said, how large is your array? If it really isn't that big, there may be something fundamentally wrong with your program

Edit: here is that tutorial, there are also examples that ship with LabVIEW

Message Edited by for(imstuck) on 06-10-2010 04:56 PM

[tbob]

The most efficient way to handle array insertion is to pre-allocate the array by using Initialize Array.  But you need to know how large your array will be beforehand.  If you know this, or if you know that it will not ever exceed a certain size, then initialize the array with some default value.  Then use Replace Array element in a loop to replace the elements.  You could always delete the default values at the end if you don't fill up the pre-allocated array with useful data.  When the array is pre-allocated, Labview does not have to make copies and re-dimension the array size.  Huge time and memory savings.

This little snippet took a small fraction of a second to replace 100,000 array elements:

[GregFreeman]

---

tbob wrote:

You could always delete the default values at the end if you don't fill up the pre-allocated array with useful data. 

---

And the opposite, if it does exceed the size you allocated you can overwrite the old data as a safeguard so you aren't indexing outside the allocated memory, but you will lose data. There are some other workearounds. Have a look at the link I posted, and at tbob's suggestion. I have seen arrays of hundreds of thousands of elements handled fine so I'm curious how big yours is getting.

[Joe D.]

Hi ed dub,

You can also use a for loop with indexing. Below is an example that creates an array with 10000 elements

Hope this helps

[Mark_Yedinak]

---

Joe D. wrote:

Hi ed dub,

You can also use a for loop with indexing. Below is an example that creates an array with 10000 elements

Hope this helps

---

For very large arrays tbob's suggestion works better. My understanding of how LabVIEW operates is that it will allocate some amount of memory for autoindexing. If you go over this size LabVIEW will need to allocate more. This will result in copying and moving the data. If the array is large enough this may happen a few times. Preallocating will ensure you have enough memory set aside and you will not need to do any additional allocation. However as tbob stated you need to know what the upper limit will be.

[tbob]

Mark is absolutely correct.  Indexing will take a lot of memory and a lot of time when the array gets large.  Indexing is great for small arrays though. With indexing, the first loop iteration will create an array of size 1.  On the next loop, the array has to be copied, re-dimensionedcto size 2, old data copied to new array, and new data inserted at end.  You can see how this will become a monster with an array size of 10,000.  With pre-allocation, there is no re-dimensioning and no copying.  Array elements are simple replaced.  And yes you do have to know the upper limit to use pre-allocation.  Kudos to Mark for pointing out the facts.

[Mark_Yedinak]

---

tbob wrote:

Mark is absolutely correct.  Indexing will take a lot of memory and a lot of time when the array gets large.  Indexing is great for small arrays though. With indexing, the first loop iteration will create an array of size 1.  On the next loop, the array has to be copied, re-dimensionedcto size 2, old data copied to new array, and new data inserted at end.  You can see how this will become a monster with an array size of 10,000.  With pre-allocation, there is no re-dimensioning and no copying.  Array elements are simple replaced.  And yes you do have to know the upper limit to use pre-allocation.  Kudos to Mark for pointing out the facts.

---

A bit of a clarification is needed. Folks from NI told me that LabVIEW tries to be smart when autoindexing. When they allocate the memory the initial chunk of memory it is larger than a single element. So you don't necessarily get a copy and allocation every iteration. They may have improved the optimization to preallocate the complete array if the size is already known (a constant wired to the loop count). However, they would have no way of doing this if the number of iterations is not known ahead of time.

[Dennis_Knutson]

There was an old post by Greg McKaskle (which I am having a hard time finding) where he explains that the compiler allocates memory in chunks for the while loop so the reallocation is not nearly so inefficient. The above example of the for loop does have memory allocated at the start and I think is just as efficient as doing an initialize array function.

[altenbach]

---

Dennis Knutson wrote:
The above example of the for loop does have memory allocated at the start and I think is just as efficient as doing an initialize array function.

---

Exactly!