DataFileLoadSel memory handling

[Phex]

Hi,

I am taking small bits and pieces from various rather big TDM-files with the DataFileLoadSel command. Strangely enough the memory consumption increases rapidly with each file being handled until it reaches almost 4 GB. Then the script stops with an out of memory error (though I still have 12 GB RAM available).

1. Is there a limitation for Diadem memory usage up to 4 GB?

2. It seems to me that Diadem is loading the whole file into the memory although I only take out a small data set. Is that correct and how can I prevent the from happening? "Load" or "Register" do not make any difference in that behaviour.

Thanks for your input.

Phex

[AndreasK]

Do you have large string channels in your tdm file?

How many values do the channels have?

Which DIAdem Version is used?

Background: DIAdem is a 32bit application so it makes use of 4GB RAM. On the other hand it is capable to swap loaded data to external swap so there shouldn't be a behavior like this. At all Load and Register should show different memory consumtion behavior.

[Phex]

Thanks for your answer.

I am running DIAdem 2012 and I only have float values in my channels with about 200 each, so nothing abnormally big. When I run the DataFileLoadSel in a loop on all my files, i can observe that the memory constantly increases proportional to the size of the whole file (although I only load 4 channels of each). At 4GB the script stops without displaying an error message and DIAdem basically needs a restart. It is not possible to work properly in any module anymore and a forced restart is required. The event viewer shows a .NET framework error (ntdll.dll) followed by the Application Error of DIAdem. I already tried a reinstallation of either .NET and DIAdem, but it did not resolve the problem.

Any idea what I could check as well?

[AndreasK]

Can you offer some more information?

• How many files
• How many channels does each file contain
• How many channels per file do you want to load
• How big is the tdm file, and how big is its related tdx file

Can you try the following:

• Settings->options->General
• Pick Data Area and set it to Swap out data : Always
• Run your script with "Load"

The maximum amount of DIAdem channels is 65000. To you strike this barriere?

Maybe putting

BulkDataLoadingMode   = eBulkDataLoadingImmediately

 as first line to your script and using "Load" may help too.

[ThomasWH · Accepted solution]

Hello,

please start the NAVIGATOR Settings dialog and switch the option "Load bulk data on modification of channel data" to "Always load bulk data" and please try again. When using TDM-files with many short channels, this setting will support your application in a better way.

Instead of "Register" please use "Load" in your case.

Greetings

[Phex]

Switching to "Always load bulk data" did it! Thank you so much.

For your information:

My biggest file I loaded with DataFileLoadSel contains 22000 channels each with about 200 values giving a size of 63MB for the TDM and 630MB for the corresponding TDX. The swap setting did not have any influence.

[ThomasWH]

Thanks for the information and Greetings