0xC0000005 Access Violation in cvirte.dll (SetWindowsErrorMode?)

[xj97]

Hello,

We will occasionally see a crash with our CVI LabWindows based 2010 SP1 application, a 0xC0000005 Access Violation exception thrown from within cvirte.dll. The app is working primarily from a spawned thread when the crash occurs. I've been able to get it consistently on a Win7 x64 VirtualBox VM and have some data that may be of use if someone there would like to investigate.

When I track down the location of the instruction pointer (work below) it seems to be in the "SetWindowsErrorMode()" function (at least going by the exports in DependencyWalker):

1. The IP is 0x687B5F1C at the time of the crash.
2. The base module address for cvirte.dll is 0x68420000, so the IP is 0x395F1C bytes into the DLL.
3. Using DependencyWalker on cvirte.dll and sorting by entry point indicates that the IP is inside of "SetWindowsErrorMode()" at 0x003938D0, which comes before "MinimizeAllWindows()" at 0x003984B0.

Coincidentally, this crash happens in CVI 2013 F1 as well, the offset within "SetWindowsErrorMode()" is a bit different (0x264C past entry in 2010 SP1, 0x2D0C past entry in 2013), but locally the code looks the same in my VS2010 disassembler:

CVI2010 SP1

*Address *Code Bytes          *Instruction
687B5F02 89 17                mov dword ptr [edi],edx
687B5F04 8B 48 5C             mov ecx,dword ptr [eax+5Ch]
687B5F07 89 78 6C             mov dword ptr [eax+6Ch],edi
687B5F0A 89 4F 04             mov dword ptr [edi+4],ecx
687B5F0D 89 58 5C             mov dword ptr [eax+5Ch],ebx
687B5F10 C7 40 60 00 00 00 00 mov dword ptr [eax+60h],0
687B5F17 5E                   pop esi
687B5F18 85 DB                test ebx,ebx
687B5F1A 74 09                je 687B5F25
687B5F1C 8B 13                mov edx,dword ptr [ebx] <--- CRASH

CVI2013 F1

*Address *Code Bytes          *Instruction
687CA2B2 89 17                mov dword ptr [edi],edx
687CA2B4 8B 48 5C             mov ecx,dword ptr [eax+5Ch]
687CA2B7 89 78 6C             mov dword ptr [eax+6Ch],edi
687CA2BA 89 4F 04             mov dword ptr [edi+4],ecx
687CA2BD 89 58 5C             mov dword ptr [eax+5Ch],ebx
687CA2C0 C7 40 60 00 00 00 00 mov dword ptr [eax+60h],0
687CA2C7 5E                   pop esi
687CA2C8 85 DB                test ebx,ebx
687CA2CA 74 09                je 687CA2D5
687CA2CC 8B 13                mov edx,dword ptr [ebx] <--- CRASH

I have dump files (mini, or mini+heap) for both builds if it would help.

This bug seems semi-similar to the thread "Is InstallPopup not threadsafe?":

http://forums.ni.com/t5/LabWindows-CVI/Is-InstallPopup-not-threadsafe/m-p/716822/highlight/true#M359...

Just for posterity, I'll add the data for the 2013 build. IP is 0x687CA2CC, DLL Module address is 0x68420000-0x68A28000, so offset is 0x3AA2CC. "SetWindowsErrorMode()" enters at 0x003A75C0 in this dll, and "MinimizeAllWindows()" is at 0x003ACB20.

Thanks-
Baker

[FoxSquirrel]

Hello,

Do you by chance have some code that can easily reproduce the error?  Is there some specific circumstances that need to occur to have the SetWindowErrorMode() generate the error, or any other specific information so we can reproduce this error internally?

[xj97]

Hi Jesse-

I do not have a smallest/simplest program built to exhibit this problem at this time. If I get some time I can try to generate one, although it was a real pain to get this to reproduce on my development machine (hence the VirtualBox VM with Win7 x64 on it).

I do have the dump files from the crash if that would help at all.

Thanks

Baker

[FoxSquirrel]

Go ahead and post the dump files and I'll pass them on.  Are you able to post the program that causes this issue on the VirtualBox VM?

[xj97]

Hi Jesse-

I'll attach the mini-dump files here. I would need to get permission to send the program exhibiting the behavior, and it would likely be a big ordeal, if it's possible at all. I could pursue that if the .dmp files are not fruitful.

Thanks

Baker

[FoxSquirrel]

Thank you for posting the dump files.  We will look into them and let you know if we figure anything out.

[K_Joy]

Hi xj97,

Can we get some more background information on when this crash occurs?

What does your application do? And, what is it doing at the time of the crash?

You said it consistently crashes on a Win7 x64 VirtualBox VM. What did you do to get it to consistently crash?

You also mentioned the application running in a spawned thread.  Are you designating a Thread Pool, or just creating a new worker thread whenever the application needs one?

Any other information you can give us about what is happening when the application crashes will help narrow down the problem.

I also have a couple of questions that will help us read the dump files:

When you reproduce this issue in 2010 and 2013 f1, are you using the same computer or using different machines?

For each dump file, which version of LabWindows/CVI were you running?  You can find the version number by selecting Help»About LabWindows/CVI within LabWindows/CVI.

Which version of cvirte.dll do you have? This will be located in System32 and/or SysWOW64.

Thanks! This will give us some context for the problem.

Kelsey Johnson

Applications Engineering

National Instruments

[xj97]

Hi Kelsey,

Apologies for the long response time, I didn't receive a notification for your post.

> What does your application do?

Our application is used in this context to setup and run our vehicle simulations on a Windows based PC.

> And, what is it doing at the time of the crash?

At the time of the crash a simulation is being launched and run on another thread from the default thread pool like this:

CmtScheduleThreadPoolFunctionAdv (DEFAULT_THREAD_POOL_HANDLE,
    sRun,
    NULL,
    THREAD_PRIORITY_NORMAL,
    NULL,
    EVENT_TP_THREAD_FUNCTION_END,
    NULL,
    RUN_IN_SCHEDULED_THREAD,
    &sThreadFuncId);

That spawned thread will load a simulation DLL, read some data as input, run the simulation, output the results, then close the DLL.

As I noted earlier, I couldn't get it to happen on my desktop, but I could get it on the VM when I used a very slow storage device for the I/O of the simulation (a slow thumb drive attached to the VM). My boss was seeing the issue on his virtualized PC on his Mac, typically only when he used a slower storage device as well (network storage for example).

> When you reproduce this issue in 2010 and 2013 f1, are you using the same computer or using different machines?

Both of these crash dumps were produced on the virtual machine as described.

> For each dump file, which version of LabWindows/CVI were you running? You can find the version number by selecting Help»About LabWindows/CVI within LabWindows/CVI. Which version of cvirte.dll do you have? This will be located in System32 and/or SysWOW64.

The development environment installed was CVI 2010 SP1. So the 2010 dump was just using the runtimes bundled with that installation. I then installed the 2013 F1 runtimes to see if the same issue persisted.

It's possible I did a clean install with the 2013 F1 dev environment, I will double check this tomorrow when I have access to the virtual machine.

Thanks for looking into this--

Baker

[FoxSquirrel]

Thank you for the information.

[FoxSquirrel]

Sorry it has taken so long to get back to you.  I wanted to check in to see if you have any new updates?  Also, I need more specific information about the build.  For example, you said you are using 2010 SP1, which means it would be 10.0.1.xxx.  I need to know what the numbers would be for the xxx.  For example, when I select Help>>About LabWindows/CVI, I get Version 13.0.0 (647).