"Building" large arrays using functional global

[Jarrod_S.]

---

@Phamton wrote:
First of all- Jarrod's suggestion to separate the "get" and "set" functions helped enormously.  Now instead of 5-6 ms per call when I have thousands of elements, it's always <0.1 ms.  So the extra copy was killing me.

However, the comments from tst and Ben now have me intriqued.  First a couple questions/comments for Ben:

---

Oh great Ben... I go and solve their problems, and then you have to go and intrigue them! 😉

[Phamton]

I think I found the answer to my last question.  I hadn't noticed the nifty option of "Return elements?" in the Get Queue Status vi.  Using this to get the array of clusters seems to have improved a test run of my program from minutes to seconds and my code is way simpler; no more  while loops polling queues needlessly.  Life is easy now   Thanks everyone for your suggestions.


Message Edited by Phamton on 01-08-2008 03:56 PM

[Ben]

"...from minutes to seconds and my code is way simpler;..."

See here for in-place.

So are we done here?

Ben

[Phamton]

Thanks, Ben.  That nugget is really making me consider switching to 8.5 with all the huge clusters I'm using.  As far as this thread code goes, I'm happy with my code now, although I'd still be interested in the answer to my question about whether Build Array in loops is as bad as advertised....

[Matthew_Williams]

A couple of comments on the 'doubling' approach:

In very general terms, this will give you an average 'array load' of 75% of an ever increasing space,
i.e. 50% at n+1 to 100% just before the next overrun.

If you allocate (current size)+10%, you average 'array load' will be closer to 95%.  You will have more
allocation events, but your final array size will be closer to what is really needed, with a maximum
over-allocation of 10% vs. 50%.  You should also be able to use more of your memory before running out.

If you can afford the overhead of allocating only enough space for the next x iterations, then your
'array load' will approach 100% as you get further out, minimizing your over allocation and
maximizing the amount of your ram you can use.

I usually create an array constant the size of the allocation chunks I want, then 'allocate' by appending
that array to my original.  I haven't timed it, but that seems to work pretty well and it should be easy
for the compiler to optimize.

Matt

[tst]

---

Phamton wrote:

I hadn't noticed the nifty option of "Return elements?" in the Get Queue Status vi.

---

Yes, I always forget about that as well, or I would have mentioned it.

---

from minutes to seconds

---

I have to admit I didn't expect such an improvement when I originally suggested this, because your global was already relatively good, but I guess it depends on the specific implementation.

Note that queues can operate in-place, but they might still create copies, depending on what you do.