Shimon,
It's not common that an application works fine in release mode and does not work in debugging. There are many different possibilities, you can try changing the size of the stack in Options>>Build options. If you built the dll the you can probably track the error down. Another thing to make sure is that you are using the right calling convention, it is possible that CVI in debug mode detects that the calling convention is incorrect, but in debug mode this is neglected.
Do you know in what language was the DLL developed?
Keep me updated!
Juan Carlos
National Instruments