How to load file with 500M into noncontiguous memory and visit them

[JWJ]

My data file consist of  a serial of 2d images. Each image is 174*130*2 butes and the total images in the file is up to 11000, which takes almost 500M bytes. I hope to load them into memory in one turn since I need to visit either image many times in random order during the data processing. When I load them into one big 3d array, the error of "out of memory" happended frequenctly.

First I tried to use QUEUE to load big chunk of data into noncontiguous memory. Queue structure is a good way to load and unload data into noncontiguous memory. But it dosn't fit here since I may need to visit any of the all the images at anytime. And it is not pratical if I dequeue one image and enqueue it into the opposite end, since the image visited may be not in sequential order.

Another choice is to put the whole file into multiple small arrays. In my case, I may load the data file into 11000 small 2d arrays and each array holds the data of one image. But I don't know how to visit these 2d array and I didn't get any cues in the posters here.

Any suggestion?

PC with 4G physical memory, Labview 7.1 and Win XP.

Thanks.

[Kevin_Price]

I'll try to get the ball rolling here -- hopefully there's some better ideas than mine out there.

1. I've never used IMAQ, but I suspect that it offers more efficient methods than stuff I dream up in standard LabVIEW.  Of course, maybe you don't use it either -- just sayin'...

2. Since you can't make a contiguous 11000-image 3D array, and don't know how to manage 11000 individual 2D arrays, how about a compromise?  Like, say, 11 3D arrays holding 1000 images each?   Then you just need ~50 MB chunks of contiguous memory.

3.  I'm picturing a partially hardcoded solution in which there are a bunch of cloned "functional globals" which each hold 1000 images of data.  This kind of thing can be done using (A) Reentrant vi's or (B) Template vi's.  You may need to use VI Server to instantiate them at run-time.

4. Then I'd build an "Action Engine" wrapper around that whole bunch of stuff.  It'd have an input for image #, for example, image # 6789.  From there it would do a quotient & remainder with 1000 to split 6789 into a quotient of 6 and remainder of 789.  Then the wrapper would access function global #6 and address image # 789 from it.

5. Specific details and trade offs depend a lot on the nature of your data processing.  My assumption was that you'd primarily need random access to images 1 or 2 at a time. 

I hope to learn some better ideas from other posters...

-Kevin P.

P.S.  Maybe I can be one of those "other posters" -- I just had another idea that should be much simpler.  Create a typedef custom control out of a cluster which contains a 2D image array.  Now make an 11000 element array of these clusters.  If I recall correctly, LabVIEW will allocate 11000 pointers.  The clusters that they point to will *not* need to be contiguous one after the other.  Now you can retrieve an image by indexing into the array of 11000 clusters and simply unbundling the 2D image data from the cluster element.

[JWJ]

Thanks for your suggestion.

Could you disclose more information about  "a typedef custom control"? Sounds it is encouraging.
The images are visited very often and usually more than 100 continuous images need to be accessed and counted into the processing result. So it is important to make multiples frames accessable at any point.

[JWJ]

I get it. Thanks a lot.

[DFGray]

Another option is to use single-element queues to hold each of your images. Store the queue references in an array. To use the image, select the queue reference from the array, dequeue the image, process, then enqueue so you don't lose the data. This method takes full advantage of fragmented memory. It also takes advantage of the fact that dequeueing a single-element queue does not cause a memory copy, so it should be fast. If you need more details, let me know.

[Kevin_Price]

DFGray:

I'm looking for some enlightenment.  I steered away from single-element queues for 2 reasons:

1. the need to handle 11000 queue references.  That somehow just "feels" excessive & I expect some overhead penalty.

2. I understand that a dequeue followed by an enqueue wouldn't tickle the memory manager, but I expect that the image processing routine(s) would probably require a wire branch that copies the image data off to a processing routine.

Comments?

JWJ: I *expect* that you can improve performance by carefully planning out an algorithm that uses a FIXED array size to store the image(s) to be processed together.  By writing this code carefully and pre-allocating the storage space, you can be sure your algorithm won't have to dynamically allocate the space over and over.

Rule of thumb for performance: data copying is not your friend, but dynamic memory allocation is the devil.

-Kevin P.

[DFGray]

Kevin,

I thought long and hard before I made that post, but I couldn't come up with any reason why 11,000 queue references would cause an issue in this case, since processing will probably be one at a time.  I agree that thrashing on all of them at once may cause problems.  I will query the real authority, Aristos Queue, and repost if there is an issue.

Careful dataflow design could result in no copies made during processing.  In this situation, I would typically dequeue the image, do the processing, then enqueue it again, flowing data through the analysis and back to the enqueue.  At the very least, the copy made from extracting the data from an array of clusters or LV2 global is avoided.  You can do the same sort of avoidance if you use an action engine as a VI template and instantiate it 11,000 times, but this overhead is far worse than the queue.

I think the choice of array of clusters or array of single-element queues is largely personal.  I don't expect there to be a huge time difference, since LabVIEW is pretty good about reusing repetitively used memory blocks.  It would be interesting to try it both ways and see what happens.

[AristosQueue (NI)]

11000 queues? Go for it.

Each queue is largely independent from the others. There is a brief critical section that only one queue can be in at a time, and that is the translation from queue refnum to actual queue, but that lasts only a few instructions and has been minimized to keep the friction between queues as low as possible. As far as memory is concerned, your cost penalty is around 10 bytes per refnum plus the size of the actual data in the queue (varies by the type of data that the queue is carrying, obviously).

There have been several apps brought to my attention that have used hundreds of queues simultaneously with no deleterious effects. I don't know of any reason why this wouldn't scale.

(PS: Notifiers are closely related to queues, and they have the same low-friction design, except when the Wait For Multiple primitive gets involved, which has to hold the common critical section longer. Just thought I'd mention that.)


Message Edited by Aristos Queue on 11-29-2007 09:11 AM

[DFGray]

One more reason to go for the single-element queue based approach over the array of clusters.  The wire for the array of clusters will be about 500MBytes.  Two copies, and probably one as well, will run you out of memory on a windows machine (max practical memory limit is 1.5GBytes).  The wire for the array of queues is 1MByte.  A few copies of this will not hurt you.

[Kevin_Price]

Thanks guys, I'm learning stuff here.  Just a couple followups:

DFGray:  ...I would typically dequeue the image, do the processing, then enqueue it again, flowing data through the analysis and back to the enqueue.  At the very least, the copy made from extracting the data from an array of clusters or LV2 global is avoided.

So it sounds like I've got some wrong ideas of how arrays of clusters are handled.  To clarify:

   (A)  Is it true that each cluster element would hold a pointer to the array it contains?  And is it further true that the memory pointed to by cluster(i) doesn't need to be contiguous with the memory pointed to by cluster(i+1)?

   (B)  When indexing an individual cluster out of the array of clusters, what gets copied?   Do you always copy the contents of the memory the cluster points to?  Or are there cases where you can you can retrieve the data as a "pointer-to-const"?

Aristos Queue:  ...Notifiers are closely related to queues, and they have the same low-friction design

I thought of Notifiers here, but only briefly because I (again) assumed that a Notifier would *have* to make a data copy.  Unlike queues, a read-like operation doesn't *consume* the data, or so my thinking goes.  Is this right or wrong?  Could a Notifer be as efficient as a single-element queue in this type of application?

Thanks!  Also, anyone else have better ideas for the original poster?  I don't mean to completely hijack the thread...

-Kevin P.