how large of a buffer can I malloc

[menchar]

Does anyone know what the constraints are on malloc'ing a buffer in a CVI application?

We run WinXP SP 2, my understanding is that there's 3GB of logical address space available for each process. But, it looks to me as if a malloc'd buffer is constrained to some percentage of physical memory.  How does the CVI runtime determine the heap size, and what rules does it use when allocating memory off the heap with malloc?

For example, if I have an application that needs a 1 GB buffer, how can I anticipate the physical memory requirement of the target system?

Thanks,

Menchar

[Ebele_Obijiofor]

Hello menchar,

According to this knowledgebase, the only limitation for the size of the heap is the amount of memory available on the system. The KB has some more information, and hopefully you find it helpful. Thanks!

Regards,

[menchar]

Thanks, but no, that doesn't answer the question at all.

In the discussion you refer to, "virtually unlimited" data storage I guess was a pun   More likely it is "virtually limited" by the span of the virtual address space for the system.  On Win32 this is usually 2 GB and maybe 3GB on XP Pro.

There is a specifc limit to what can be malloc'd in a Win32 application, you can easily prove this by trying to allocate an arbitrarily large buffer.  On my XP system, with 1GB of physical memory, and a paging file of 4GB, I can allocate a 1.5GB buffer, at least some of the time.

I would guess that the CVI rte uses the process heap in Win32, rather than a private heap.  So, arguably, this is more an OS question than a CVI question, unless CVI imposes some additional constarints that aren't stated anywhere.

We deal with very large video pixel arrays and I'm trying to develop a sense of how I can deal with them.

Thanks for your response.

Menchar

[LuisG]

Hello menchar,

The theoretical buffer size limit in a 32-bit app is 2Gb. At least that's how much you can allocate in a simple "hello world"-type application, in Visual Studio. In CVI, however, the limit seems to be around 1.5 Gb. Although CVI has its own memory manager, its memory manager is not imposing any artificial constraints on how large a buffer size you can allocate. However, in investigating this issue we have noticed that once the full set of libraries that the CVI run-time requires is loaded into the process, we found that the OS no longer allows 2Gb allocations. This seems to be related to the overall amount of code that is loaded in memory, and not to anything in particular that this code is doing.

If you're interested in allocating very large buffer sizes, my advice is to wait until a native 64-bit version of the compiler is available.

Luis

[menchar]

Luis -

Thanks for the reply.

Yup, 1.5GB seemed to be the limit, as you say this could be the simple consequence of loading the various CVI libraries, the process, stack, etc.  against a 2GB address space.

I don't think WinXP SP2 goes to 3GB process address space by default, so the behavior could change if it's working against a 2 GB address space now.  If the OS is providing 3GB and I can still only get a 1.5GB malloc, I would of course wonder why, but I'll track this down some more.

The risk is that the process's behavior is such that it breaks for size during runtime when anything tries to malloc additional memory off the heap.  The stack size comes into play as well.

Sure, a 64 bit OS changes everything, but we want to get as much mileage as we can out of our current systems.

Thanks again, I'll post anthing more I find out (e.g. try forcing 3GB address space) to this thread.

Menchar

[menchar]

More info on buffer sizes.

When I set the "/3GB" switch in the boot.ini file on my WinXP Pro SP2 system, it failed to completely initialize, rendering the system unusable.  I could do a three finger salute and get the security popup to appear, but it wouldn't run the task manager or do anything else.  Cursor was active, wallpaper appeared but that was it.  So the system was trying to go to 3GB but wasn't getting there completely - don't know why.  I had to do a safe boot to get back at the boot.ini file and remove the 3GB switch.

System has 1 GB of physical memory, 4 GB backing file.

I did not use the PAE (physical address extensions) facility since I had just 1 GB of physical memory installed.  My understanding is that I should have been able to use a 3GB logical address space without using PAE.  I couldn't find anyone here at work who had used the 3GB switch without the PAE also enabled.  I didn't see any mention in the MSDN KB about a minimum physical memory needed to enable the 3GB logical address space.

The executable image needs a bit set in its header to make it "large address aware".  The bit is normally set by the linker - it's not clear that the CVI linker can be made to do this.  In any event, you can patch the image header using a separate utility if need be.

Menchar

[LuisG]

Menchar,

As of version 8.1, the CVI linker does mark its executables as "large address aware". (Of course, that won't help you much if you can't booth with the /3Gb option... )

Luis

[Gcyin]

Hello,

Just share some experinces. I have one server that has the 32G bytes in physical memory with the operation system (winserver 2003 64bit), and  I have tried to allocate the memory as large as possible with CVI. The maximum memory I can allocate is about 2G byte. Is that the limitation of the CVI itself? because it is not a 64 bit compiler? The Version I use is 7.1.1.  Is there any way I can allocate more than 2G?

Gung-Chian Yin  

[LuisG]

My understanding is that in normal circumstances, a 32-bit version of Windows allows 4Gb to be allocated per process, but of those, 2Gb are reserved for kernel mode. Which means that you can only allocate 2Gb in user mode (CVI apps run in user mode).

There is a Windows option, that you can specify in boot.ini, called /3Gb. With this option you can, in theory, expand the virtual memory in your process' user space to 3Gb, but in order to take advantage of it in a CVI application, you need to build it with CVI 8.1 or later.

Luis

[menchar]

That's a very good question.

Luis says the "theoretical limit" is 2GB, but I believe it should be more than that if you are using the /3GB option of WinXP Pro or the other 32 bit Windows OS's that support the 3GB user space option.  I would have to think the heap size of a 64 bit process on the Windows 64 bit OS's is very much greater.

I am sure someone from NI can tell us what the 64 bit situation is with CVI and the process heap specifics.

I believe the native CVI compiler is an adapted LCC 32 bit compiler which you can read about here:

http://www.cs.princeton.edu/software/lcc/

So unless NI has changed this or unless you're maybe using an external compiler, I'd guess you're running 32 bits only.