What is the most efficient way of passing large amounts of data through several subVIs?

[UHB]

I am acquiring data at a rate of once every 30mS. This data is sorted into clusters with relevant information being grouped together. These clusters are then added to a queue. I have a cluster of queue references to keep track of all the queues. I pass this cluster around to the various sub VIs where I dequeue the data. Is this the most efficient way of moving the data around? I could also use "Obtain Queue" and the queue name to create the reference whenever I need it.
Or would it be more efficient to create one large cluster which I pass around? Then I can use unbundle by index to pick off the values I need. This large cluster can have all the values individually or it co
uld be composed of the previously mentioned clusters (ie. a large cluster of clusters).

[Dave_Kaufman]

If the data is all of the same type, use arrays instead of clusters. Also, use an array for the set of queue references. Clusters require tremendous overhead in LV.

[Ben]

Hi UHB,

I believe the most correct answer is DON'T!

Check out the solutions to past LV code challenges at http://www.ni.com/devzone/lvzone/codingchallengearchive.htm.

Also check out Dr. VI's articles about moving data.

Summarized.
Moving large amounts of data takes a long time! It should be avoided wherever possible. This can be done by acting on data in place. When you collect the data, put directly into the object or objects that will be using that data. That way the data gets moved once. From that time forward, you perform in place operations.

So...
If you want to go faster, do less along the way.

Ben


Ben Rayn
er
Certified LabVIEW Developer
www.DSAutomation.com

[Greg_McKaskle2]

> I am acquiring data at a rate of once every 30mS. This data is sorted
> into clusters with relevant information being grouped together. These
> clusters are then added to a queue. I have a cluster of queue
> references to keep track of all the queues. I pass this cluster
> around to the various sub VIs where I dequeue the data. Is this the
> most efficient way of moving the data around? I could also use
> "Obtain Queue" and the queue name to create the reference whenever I
> need it.
> Or would it be more efficient to create one large cluster which I pass
> around? Then I can use unbundle by index to pick off the values I
> need. This large cluster can have all the values individually or it
> could be composed of the previously mentioned clusters (i
e. a large
> cluster of clusters).

It sounds pretty good the way you have it. In general, you want to sort
these into groups that make sense to you. Then if there is a
performance problem, you can arrange them so that it is a bit better for
the computer, but lets face it, our performance counts too. Anyway,
this generally means a smallish number of groups with a reasonable
number of references or objects in them. If you need to group them into
one to pass somewhere, bundle the clusters together and unbundle them on
the other side to minimize the connectors needed. Since the references
are four bytes, you don't need to worry about the performance of moving
these around anyway.

Greg McKaskle

[GabbyG]

I have real huge sets of data originating from an image-like (many "pixels") data aquisition that is interpreted according to a similar large set of objects with many properties. This is all fed also through different GUI windows.
Before I had this fed into and out of the sub vis via ontrols/indicators as learned.
Now I just finished converting into keeping the master set of data in gobal variables and creating a local copy at the entry of the sub VI.
Background of this not so common approach is, that once a front panel was open ( as in a user interface happens eventually) the data in the controls/indicators is still using up memory, Whereas since LV 7, I can release local memory like copies of global variables on exit.

Of course in a man/machine inte
rface speed is not so essential and timing performance may require a different approach, but my problem was reaching the all terminating limit of 2GB / process.
Linux 64 with lots of RAM might have helped otherwise 😉
Gabi