Need to allocate 32Gb of memory with NI-Vision module (VisionImage)

[BruceAmmons]

Rafiou,

Can you explain why you need a single image that is 32 GB?  I don't understand why it can't be divided into sections and analyzed one section at a time.  What kind of analysis are you doing that won't allow subdivision of the image?  Perhaps you just need to rethink your approach.

Bruce

[Andrey_Dmitriev]

---

rafiou wrote:

That means the 64-bit LabView is not a complete 64-bit platform. It is a quite disappointing that we can not enjoy the full capacity of a real 64-bit application with Vision. Even though you have VDM in a LabView 64-bit, NI is still deploying 32bit vision components on it.

---

Hmmm... Here not sure, because NI said:

"Q. Which LabVIEW modules, in addition to NI Vision, are available for 64-bit PCs?
A. The NI Vision Module is the only module available for 64-bit PCs."

 (see link above)

Otherwise, even if you will be able to allocate 32 GB image, then don't forget, that you will need 32 GB continuous memory. If your memory is fragmented, then you will never get such big piece. (For example, if you have 1 GB Free memory, then try to allocate 1000 of 1 MB images, then delete every second image, then try to allocate 500 MB Image - you will be surprised).

So, I agree with Bruce - you should split your image to small pieces.

Andrey.

PS

"LabView" - written incorrect. The only "LabVIEW" is correct. 😉

[rafiou]

 Dear Bruce,

I am acquiring high resolution images from a slide with a microscope and a moving stage that does not move perfectly horizontally and vertically. So, I have already developement a stitching algorithm (based on the mattching pattern) that stitch each images (acquired with 10% overlapping horizontally and vertically to correct for any misteps from the stage and proper stitch).

It works perfectly, but I can not process by dividing the images by section but the resulted stitched image is not perfectly horizontal and vertical. I will not know where I am the image if I do that. 

Thanks

R

[rafiou]

Dear Andrey,

I am really aware about the possible fragmentation issue but the virtual memory can act as if we have a large contiguous physical memory. As it said the "Background on Computer Address Width and Memory Management" section in this link: http://zone.ni.com/devzone/cda/tut/p/id/10184 

May be I should re-think my approach too as it will be more carefull.

Thanks.

Rafiou

I take into note this time. It is  "LabVIEW" and not LabView". I am not very familiar with LabVIEW yet.

[Andrey_Dmitriev]

---

rafiou wrote:

Dear Andrey,

I am really aware about the possible fragmentation issue but the virtual memory can act as if we have a large contiguous physical memory. As it said the "Background on Computer Address Width and Memory Management" section in this link: http://zone.ni.com/devzone/cda/tut/p/id/10184 

---

Thanks for this note. Sounds interesting, I'll check it on 64 bit PC this evening.

Otherwise, NI wrote:

"...Vision Development Module 64-bit is intended for applications that require large images (100-200 MB). Displaying and processing large images requires several large additional image buffers to be available simultaneously, and in 32-bit operating systems, this can result in error messages due to insufficient buffer in the OS, even in systems with 2-3 GB RAM.  
The new 64-bit edition of the Vision Development Module circumvents these memory issues with its native support for 64-bit operating systems for Windows Vista 64-bit and Windows 7 64-bit.  The 64-bit edition of Vision Development Module 2009 allows applications to operate on images of up to 2GB in size..."

Andrey.

[rafiou]

Dear Andrey,

Thanks again for checking. As you can see it is interesting, VDM limits to 2GB but LabVIEW array does not. The overall thing and the most interesting part is normally in a real 64-bit application with a real 64-bit computer equiped with the neccessary physical memory, allocating 12GB or 32Gb is jus peanut compare to its full addressing capacity.

Thanks,

Rafiou

[BlueCheese]

---

Andrey Dmitriev wrote: 

Otherwise, even if you will be able to allocate 32 GB image, then don't forget, that you will need 32 GB continuous memory. If your memory is fragmented, then you will never get such big piece. (For example, if you have 1 GB Free memory, then try to allocate 1000 of 1 MB images, then delete every second image, then try to allocate 500 MB Image - you will be surprised).

---

Hi Andrey,

Vision images must be virtually contiguous, not necessarily physically contiguous. Thus the amount of "free" or "total" memory on the system is pretty inconsequential. On a 32-bit process you are correct that large blocks in the 100MB+ range become pretty scarce because the memory space becomes fragmented. However, for a 64-bit process the virtual memory space for a process is 8TB I believe, so it would be fairly hard to get that space fragmented enough that even a request as large as 32GB would fail [ignoring the pain of swapping to disk if you don't have enough physical memory to back that up...].

Eric

[BlueCheese]

---

rafiou wrote:

Dear Andrey,

Thanks again for checking. As you can see it is interesting, VDM limits to 2GB but LabVIEW array does not. The overall thing and the most interesting part is normally in a real 64-bit application with a real 64-bit computer equiped with the neccessary physical memory, allocating 12GB or 32Gb is jus peanut compare to its full addressing capacity.

Thanks,

Rafiou

---

Rafiou,

While you are right that 64-bit processes ultimately should not be running into addressing limits, it is not just a simple matter of addressing. There are also internal code paths that have to be considered and guaranteed that all operations they do can work with such large sizes. This is especially a concern with regards to backwards compatibility and code compatibility between 32-bit and 64-bit code.

For instance, while you are right that LabVIEW does not limit arrays to 2GB, they are still currently limited to a signed, 32-bit integer for each array dimension. Thus, a single 1D array of U8 values is still limited to 2GB. My understanding is that this tradeoff was made (for now) in the interest of backwards compatibility because there is a lot of code (both internal and external) that expects array dimensions to be signed 32-bit values. For instance, VI server can share data and make remote calls between 32 and 64-bit versions of LabVIEW. If array dimensions can be 64-bit, you start running into issues sharing data between the two.

Another example is in terms of APIs. For instance, many VDM processing VIs return results that make references to counts of pixels. If you can have more than 2 billion pixels in an image, the return value of that VI must now be promoted to a 64-bit value (these have been 32-bit before). This is an interface change that must be done in a backwards compatible way so that users can run the same LabVIEW code on both 32-bit and 64-bit LabVIEW.

[side note: C APIs typically solve this type of thing with a pointer-sized integer that varies in size based on the size of a pointer in that process. LabVIEW does not currently have such a variable-sized datatype on the diagram which complicates such matters - don't ask me why this is...]

So while yes, in an ideal world it would be nice if everything LabVIEW and VDM had no limitations other than pure address-space limitations, but the reality is that we cannot deliver every feature that every customer desires [and have it available yesterday 🙂 ], especially in the first 64-bit release. If you consider simply the comparison to the 32-bit version, it still has many huge advantages that will help many, many customers. For instance, customers with a 5 megapixel camera would be lucky to get a couple hundred image buffers in memory before hitting limits in a 32-bit process. With some cameras that might only be a few seconds of data. With 64-bit LabVIEW and VDM they are limited only by the memory they have in the system. So while you are correct that the 64-bit version of LabVIEW and VDM is not yet ideal for customers like you that have single, huge images (larger than 2GB) or arrays (larger than 2 billion elements per dimension), there are still many application areas that it helps.

Eric