Memory management - Data area

[mdelsol]

Hi,

I need to define a data area with 65 channels and a channel length of 10000000. With different configurations DIAdem shows up the following error messages:

1-The defined area is too large for this computer to reboot

2-Numeric value out of valid range (6400000000>2147483647)!

Error occurred while installing the data matrix

Does anyone know how to solve it? Minimum necessary memory RAM? Free space necessary on HD? I tried with DIAdem 8.1 & 9.1 with same result.

Help will be appreciated,

Thanks.

[Ingo_Schumacher]

Hello,
you have to know that each value in the data area is represented as a R64 value - that means each value takes at least 8 byte (its a bit more in DIAdem 9.1). Talking about 65 channels and 10 Million values each that will make about 4.7 GB.
Surely you do not have enough RAM to store all this values (plus the memory the OS and running applications take) - that means you need to create a swapfile.
Depending on  your operating system and the way it is configured, the filesystem you are using, and the empty space on your harddisk there might be some limitations to the sice of data which can be swapped to the disk.
Try to increase the maximum swapfile size in your windows settings.

If you are using DIAdem 9.1 and you need the large data area to be able to load and display large datafiles, you might like to try the option to register (I am not sure about the english term, it is called "anmelden" in the german version) file instead of opening it. This will leave the content on the harddisk. But you still can display the data and work with it - even though the channels are flagged as read only.
It is very much faster because you do not need to swap the RAM to a file.

regards

[Brad_Turpin]

Hi mdelsol,

So, what are you planning to do with 65 channels, each having 10 million values?  Just plot data?  Run an FFT?  Look for trends?  Depending on what you want to do, it may not make sense to hold all that data in memory at the same time, which your operating system by itself is currently not able to do anyway.  There are several ways of doing that.  First, you can only load the data channels you need at a given time by using selective loading-- DIAdem will happily load only a subset of the channels in a data file.  Then you can stage your analysis or report creation sequentially, releasing the data memory from stage 1 before proceeding to stage 2.

Alternatively, it may make sense for you to use the "Register loading" feature, where the data file is used as in-place virtual memory, managed by DIAdem.  DIAdem 10 graphs registered data WAY faster than DIAdem 9.1, so if this solution looks interesting, I highly recommend trying it with an evaluation copy of DIAdem 10.

Finally, it sounds like you are currently configuring static channels in DIAdem's memory matrix in anticipation of these high bandwidth operations in DIAdem.  This was absolutely the correct approach in DIAdem 8.1 and earlier versions, and in some cases also in DIAdem 9.0.  But with DIAdem 9.1, your channels are now managed completely dynamically.  DIAdem 9.1 will automatically request memory from the operating system (RAM and virtual memory) as needed, and when the OS gives up at 2 GBytes, DIAdem will manage the rest of the memory in its own HDF5 virtual memory layer.  All seamless to the user and under the hood.  Note though, DIAdem-managed virtual memory, like OS-managed virtual memory, is not fast...

Let us know what sounds best,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[mdelsol]

Thanks for your quick answer,

Actually I solved the problem splitting the Data files to reduce the length and after the analysis I join the results channels. I did it in this way because I didn’t want to expend much time modifying the code that I already had to analyse the data. My code performs several filters to the data and then searches all the positive peaks over zero and the negative peaks under zero in order to plot some histograms related to the peaks.

I think that your solutions are well suitable but I thought that how I already had the code that was possible to generate that memory matrix…

Now just for my knowledge: It’s possible to generate that data matrix with 1GB RAM memory? The problem could be that due to my company policy our HD are in FAT32? If I use a NTFS HD then it’s possible?

Thanks for your interest,

Marc.

[Brad_Turpin]

Hi Marc,

I'm glad to hear you have a working solution.  Splitting the file up or just loading selected channels from the file is an excellent solution to this bandwidth issue.  In answer to your last question, the File Allocation Table does affect the maximum amount of addressable RAM as well as the maximum file size on disk:

FAT       2 GBytes (2^31 bytes)
FAT32   4 GBytes (2^32 bytes)
NTFS    effectively unlimited

None of this, however, affects how DIAdem 9.1 manages memory, particularly the fact that DIAdem's memory management with version 9.1 and later is completely automatic.  DIAdem will rely on the operating system and available RAM for as long as possible, but when that finally gives out DIAdem 9.1 will seamlessly shift over to managing the rest of the memory with its own virtual memory layer.  If you have lots of RAM and a File Allocation Table that allows all of your RAM to be addressed, then the operating system will of course take you much further, and DIAdem doesn't have to bother with its own virtual memory management.  Note that under "RAM" in the previous two sentences, what is really meant is "RAM plus operating system virtual memory".

So, it is true that with 65 channels each 10,000 rows long that your memory requirement (5.2 GBytes) far exceeds what an operating system with a FAT file allocation table can reference (2 GBytes), but this shouldn't really affect the way DIAdem behaves as you cross that boundary.  Just create the (dynamic) channels normally in DIAdem as you need them, and DIAdem will manage the rest (as long as you don't try to manage it yourself manually with static channels).

Clear as mud?
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[mdelsol]

Brad, thanks you very much for the explanation, as you ask now it's clear as mud, but much more clear than before...