This MSDN document describes those various options. Im not sure why you would be getting these kinds of failues, I would recommend posting on the
LV forum to see if someone has seen this before.
That error is an indication that something generated an error in your DLL. One thing you can do to track such issues is to debug into the DLL using Visual Studio. Under the debugging options in the project settings, set LabVIEW.exe to be the external process. when you run the Visual Studio project, it will launch LV for you. You can set breakpoints in your DLL code and when you run the VI, the debugger will let you step thru the DLL code.
One thing to mention is that if you are allocating memory for data that you want to pass back into LabView, you need to use the LabVIEW memory manager functions. Using malloc and calloc will not work, since these are not labview aware. This also depends on how you have configured the "Call Library Node". You can find more information about using DLLs with LV in the
Using External Code in LabVIEW manual.
Hope this helps
Bilal Durrani
NI