dynamic array takes very long to compile

[ISITEC_RIEPER]

Hi there,

in a small programm i need to create a large array:

FIXED SIZE ARRAY:
char hashtable[27][26][999][99][16];

But not in all cases the last element of the hashtable has a value (string
of 15 Byte) , so i want to create a dynamic array to save memory.

DYNAMIC SIZE ARRAY:
char *hashtable[27][26][999][99];
.......
if (strlen(hash_string)) hashtable[a][b][c][d] = malloc(sizeof(char) *
(strlen(hash_string) + 1));

.......


In debug mode:

If i compile the program with the fixed size array everything works fine and
the program runs immediatly

If i compile the same program with the dynamic size array (and using malloc)
it takes really really long to compile. It looks like the program freezes,
but if i wait long enough (many minutes) the program runs


I'm using CVI 2009

Any ideas why it takes so long if i use dynamic memory?

Thanks in advance,

Norbert Rieper
Bremerhaven

[Jan_Wagner]

Hi Norbert,

right now I'm not sure what leads to this behavior but you can try two things.

1. What happens in release mode?

2. What happens if you use calloc instead of malloc? (see link)

How Does Dynamic Memory Allocation Work?

http://digital.ni.com/public.nsf/websearch/119E4AA17E0A6F9C86256C4E00568121?OpenDocument

[tstanley]

If you have the Debugging level set to Extended in the Build Options, I know the debugger does some extra stuff when it comes to dynamically allocated memory.  That could be what's slowing down the build.

[Michael_H.]

Norbert,

Assuming you are iterating over all the elements in the 1st four vectors of the array then you are potentially calling malloc 69.4 million times. This will always be tremendously slower than 1 large allocation. There are also a few bytes of overhead for each pointer that you allocate so you might not be saving any memory unless your hashtable is almost empty.

Michael

NI

[ISITEC_RIEPER]

Hi,

thank you all for trying to help!

@niceday:

1. I tried to create a exe file in release mode and it results in fast compilation. It takes only a few seconds to compile the program and create the exe file.
2. To use calloc instead of malloc doesn't change anything. I can also delete the whole code of my project and only create:

char *hashtable[27][26][999][99]

int main (int argc, char *argv[])

{

}

and still the compilation hangs in debug mode and works in release mode.

@tstanley:

The Debugging level is set to standard in the build options.

@Michael H.

if i would do 1 large allocation i also could use a static array. I know that the array has about 69 Million elements but i also know that i have to malloc memory for only 200000 strings (the dynamic element) within the array. So if the string has about 16 Bytes i need only about 70 MByte Ram. With a static array this would be about 1GB Ram.

In summary the problem is not the part of code where malloc or calloc is used, the problem ist the line where the array is defined (created).

Any other ideas?

Thanks from Bremerhaven, Germany

Norbert Rieper

[Michael_H. · Accepted solution]

When CVI compiles the file in a Debug config with the debugging level set to Standard or  Extended then we allocate user protection data to track the state of each pointer in the array. For each of the 69 million pointer values we are allocating data to be added to your executable for run-time checking. These allocations are causing the apparant hang.

There are multiple ways to prevent CVI from allocating all of this additional data. 

1) Change the debugging level to 'No run-time checking'. 

2) Change the base type of the array from 'char *' to 'size_t'. You would also need to add a cast to every reference of the array elements.

3) Select the 'Enable .Obj option' option for the file that contains the definition of the array. This disables all debugging for that .c file.

3b) Move the declaration of the array to it's own .c file and select the 'Enable .obj option' that that .c file instead of the .c file that contains all of the source that uses the array.

Even if you only need 70 megs for the dynamically allocated strings, you still need 265 megs to store the array of 69 million pointers.

-Michael

[ISITEC_RIEPER]

Thank you Michael,

i think i take the 'Enable .Obj option' 

Norbert