Well, the error message is generated by LabVIEW if any SEH exception is thrown from the DLL, so something inside your DLL is having problems. This could be an access violation (NULL pointer access, bad address, etc), out of memory, etc. Any type of exception that could be thrown either by you or the system.
Assuming this is a Windows system and you have the source and VS debugger, you can tell VS to break whenever any exception is thrown.