How to load file with 500M into noncontiguous memory and visit them

[AristosQueue (NI)]

Your question about the notifiers is correct. The "Wait for Notification" behaves like the Preview Queue operation. The data is copied so that other waiters can get a copy of the data later. The queues are what you want.

[JWJ]

Wo,. so many replies. Thanks a lot first.

I report what I did and observed here.

I bundle each 2d image into a cluster and make an cluster array with dimension size 11000.  The whole cluster array takes almost 500M memory, which uses the noncontiguous memory since it will be out of memory otherwise in my computer. There are 2 copies of this cluster array during the runing, one is the original array, another is not a real copy, it is only the same size of memory space taken by labview since I put the data loading in a CASE structure. So there are 1G memory taken out of the CASE. Now I can access each frame of image. Compared to the 3d array, the only thing not straightforward here is I can not access the 11000 numbers on each pixel with one function.

Kevin, I changed the dynamic allocation to initiate the clust array outside the image loading in the For loop, as your suggestion.

DFGray, thanks for your suggestion. I may not try the QUEUE method. But I just wonder how to put the reference to each QUEUE into a one dimention array by programming. The only way I know to make reference (to the QUEUE) is  to output the QUEUE into a indicator and create a reference manually then.

[DFGray]

JWJ,

I attached a simple VI which shows how to efficiently make the array of queue references and how to access your data once you have the queue.  If you use this method, it could save you 500MBytes of RAM.

In answer to the earlier "thread hijack" question, storing data in an array of clusters does break it into noncontiguous memory space, but does not reduce the size of the wire itself.  If the wire splits or is copied, the entire data structure is copied, as JWJ found out  The array of queue references, on the other hand, is essentially an array of pointers (I32s on WinXP).  It will act very similarly to an array of pointers in C/C++.  If you copy it, you only copy the pointers.

Can you extract data from an array of clusters of data without making a copy?  Maybe... sometimes...  The newer your copy of LabVIEW, the easier this is.  The new In Place Element structure in LabVIEW 8.5 makes this particularly easy.  However, JWJ is using LabVIEW 7.1, so the array of queues is much safer.  It works in any version of LabVIEW where the queue supports arbitrary data (LV5.1 and later???).

[JWJ]

Hi, DFGray,
When I use cluster array, I can put them in the self register, under this way I can load the data only once and use the data in memory across the analysis where I may stop the program and run it again at anytime. This is the reason I use the CASE structure to load the raw data if it is not in memory and avoid reloading if it is. It could reduce the copy of memory with the use of 11000 QUEUEs. But how to set up to avoid the reloading whenever I have to stop the program in the middle of analysis and try to run it again later? Can the QUEUE not be released when I stop the program and reuse it?

[AristosQueue (NI)]

The queue is a pure run-time construct. Once the VI stops running, the queue is deallocated. In order to keep the queue around, you'd have to have another VI that opened a second reference to the same queue (this means you have to give the queue a name), and have that other VI keep running even while the first is stopped and edited.

[JWJ]

I see. I guess so when I test my code here. Thank you.