allocating +1GB of memory with one array

[demux]

Hey all,

I received this bizarre yet interesting question from a customer of NI.
The guys who have this 'problem' are all impressive programmers and they were interested whether we knew a reason or a solution/workaround. Or that they simply stumbled on the boundaries of windows/labview/PC system in general.

They are working on a huge 3D rendering project and they want to allocate as much RAM (with one array) as possible for their calculations. Having to do them from harddisk really slows down the process x100. They also made sure that they didn't programmed it in such a way that LabVIEW needs to make a copy of the array at some point for calculations.

On a pc with 1GB of RAM, they can get upto around 400MB. Trying to allocate more at once results in an 'out of memory' popup when the initialize array.vi executes. This 400MB is also not system dependant, every pc (with 1GB of RAM) they tried (3) can reach about the same amount.
All the memory is not in use at this point according to the windows Task Manager.

On a PC with 4GB of RAM however they can hardly reach 1GB. Same message, but again the RAM cannot be 'really' full at this point.
We presume it has something to do in the way LabVIEW (or windows) allocates the memory. For instance that LabVIEW searches for one complete window of 1.5GB of free RAM instead of 2 windows of 750MB for instance, 10 windows of 150MB... therefore not finding a big enough free window (1.5GB) and coming up with the out of memory popup. Because splitting up the array indeed enables more memory to be allocated, they've tried that but it's not extremely efficient.


Any help, suggestions, feedback about possible reasons is appreciated. Solutions/workarounds would be even better. Any suggestion that can help allocating more RAM (without splitting the array into partitions) is welcome.


Thx,
Wouter

[unclebump]

What type of data are they assigning to the array?? Dbl, Sgl, I32??

[demux]

It's a 2D array of singles

[DFGray]

Check out this tutorial, Managing Large Data Sets in LabVIEW, for a way to allocate one copy of data and access it anywhere without making copies. That said, Windows does not like to give a single process more than about 1GByte of RAM. Using the techniques in this article, I have successfully acquired a 1GByte buffer on a machine with 512MBytes of RAM. Yes, it got into my virtual memory. Yes, it was very slow. No, I did not get an out of memory error. However, 1GByte is probably the practical limit until 64-bit processors, OS, and LabVIEW are available. You will be able to do a bit more than that, but not much.

[eyaly]

Hi,

Have you found a way to open your array ? I am now confronting the same issue - reading a 1.1GB array to memory, but could not do it yet. I have managed to open a 0.5GB array and process it successfully.

I will be glad to share ideas with you,

Thanks,
Eyal

[DFGray]

You are running into the limits of Windows/LabVIEW. You have a maximum of 2GBytes you can use, total, for everything. If you allocate a single array in LabVIEW, it does so in contiguous memory. If your memory is fragmented, you won't get anywhere close to 2GBytes before "running out". Try allocatiing several sub arrays to increase your total. I can usually get to about 1.4GBytes this way.

[XCoop]

DFGray-

Is Windows or Labview the bottleneck on the memory? If it is windows, can't you set the memory available for windows to work in. I know One of my applications (not one I have written) has that option. My machine has 3Gb of capacity and I typically set W2k to reside in say 512Mb and use the rest to stream highspeed images to ram and later save to disk. Curious if you could do something similar in LV if it lets you go over a 2Gb of ram

XCoop

[shoneill]

Well considering that a 32-bit integer can address a maximum of 4 GBytes, the limit is going to be an OS one at some time or another. Windows 64-bit is around the corner, so maybe there'll be an easy solution soon.

On a side note, why is 2 GB the limit? Does Windows reserve one bit of the address space for something else?

Shane.

PS This kind of reminds me of an old Intel TX-motherboard chipset (BX? I can't quite remember) which could hold something like 2 GB, but could only actually address 512 MB of it. That was so silly, it was cool.

[unclebump]

Here is another thread on this topic.

http://forums.ni.com/ni/board/message?board.id=170&message.id=120840&view=by_threading&page=1

[DFGray]

XCoop,

My understanding is that Windows runs system code in the negative memory space and everything else in the positive. As such, it would not surprise me that there are ways to get to the negative space that allow you to break the 2GByte barrier. However, to the best of my knowledge, they are not supported natively in LabVIEW. Nothing would prevent you from writing a DLL to do such a thing and calling it from LabVIEW or using some system utility to configure it. I have done similar things using external DLLs, and they worked well (specifically, I used HDF5 as a RAM buffer).

I must admit, however, that I am NOT a Windows programming expert. I usually go find a real expert when I have questions like this.