Lookout 6.5 critical crash - writes to random modbus addresses

[T W]

Hello,

We are experiencing a rather critical problem with Lookout 6.5. The proces file contains approximately 148 Modbus/TCP connections, some overview panels, a properties panel which uses a symbolic link to retrieve properties from a certain modbus object.

Seemingly random, Lookout crashes during operation (with the Windows send/don't send window). When this happens, the second before the crash actually occurs Lookout starts writing seemingly random values to addresses to all the modbus devices. Some to addresses which are connected in Lookout, and to some that are not (even to addresses that actually don't exist on the device, which causes it to return an exception). The addresses do seem to be in an increasing sequence (i.e. it starts at 500, then starts writing further until about 850). Values I've seen are mostly 0, but also 256, 2308, 512, 2048, 3844.

I've attached a packet capture log which shows the write commands Lookout is sending.

Does anyone have any idea what could cause this crash or things I can check?

[Ryan.S]

Please follow the steps to create a memory dump on the crash.

Start->Run, input

drwtsn32 -i

Start->Run, input

drwtsn32

Configure the Log file path like c:\crash, and the crash dump file like c:\crash\dump.dmp

Upload the log file and the crash dump to ftp://ftp.ni.com/incoming

Please also upload your lookout process to ftp.

[T W]

Hello,

I've uploaded the data as "Lookout random writes.zip". Please also see the included readme file.

[Ryan.S]

Got the files. From the drwatson log, the crash does happen in the Modbus driver. I'm trying to reproduce it by your process.

But I still want you to try to generate the dump file. When you configure the Dr Watson, do you specify the dump file name? The log file is just a path, but you need to give a file name for the crash dump.

[rob_v]

Update test results; with no Ethernet connection or only one modbus connection the system runs without crash.

2 or more (identical) modbus connections --> crash within 1 hour. (random between 5 minutes and an hour).

Process stable when setup as read only. (no URL connections).
With URL connection, multiplexed by a symbolic link ("..\Modbus"&text(Pot_show_Turbine,"000")

Dump is setup in drWatson, including file name, but somehow didn't do anything. Maybe restricted directory, so set path to root directory.

Waiting for crash, hope to send dump as soon as possible.

Please understand that a super large project is completely on hold, so we working day and night. Hope you can save our ash

[Ryan.S]

Could you give me the process with just 2 modbus connections which will crash in 1 hour?

I want the process as simple as possible. It will be easier for us to reproduce and debug. Thank you.

[rob_v]

We used the identical process, just unplugged the ethernet cables and left one, or two or more. For test it schould be enough to change random a couple of ip numbers to whatever you setup.

[rob_v]

Trying to make a dump; didn't succeed. Made a test crash, witch made a dump.

Also detected something new; suddenly modbus starts to write random addresses with random content. We saw that already at the same time as a crash, but this time it didn't crash, just did the random write.

As attachment the capture file from wireschark, filtered on write commands. Maybe it gives another idea. For us it answers the question; is random writing a result of the crash, or is the crash a result of random writing. Half answer; random writing is not a result of the crash, not 100% sure if it results in a crash

[T W]

I've uploaded a dump file as "Randomwritecrash.zip" - it wasn't made with Dr. Watson because that doesn't seem to work at all, I've made it with WinDbg (attached to process; after crash I ran .dump /f file.dmp). I hope you'll be able to use it.

[T W]

Also uploaded a second dump file as Randomwritecrash2.zip - I'll keep uploading dump files in sequence.

Crash data from this dump:

(1328.bd4): Access violation - code c0000005 (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

eax=01cbcd1a ebx=00f70000 ecx=3e720012 edx=3e720013 esi=00f70838 edi=00000009

eip=7c902a9d esp=0013f4f4 ebp=00f70838 iopl=0         nv up ei pl nz na po nc

cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010202

ntdll!tan+0xcf:

7c902a9d 8b18            mov     ebx,dword ptr [eax]  ds:0023:01cbcd1a=????????

0:000> kb

ChildEBP RetAddr  Args to Child              

WARNING: Stack unwind information not available. Following frames may be wrong.

0013f4f8 7c91020e 00000009 00f70838 00f70000 ntdll!tan+0xcf

0013f510 7c9101db 0013f744 7c90e920 7c910228 ntdll!RtlAllocateHeap+0x14a

0013f528 7c91019b 00f70838 00000013 0000003f ntdll!RtlAllocateHeap+0x117

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\WINDOWS\system32\MSVCR71.dll - 

0013f754 7c3416b3 00f70000 00000000 0000003f ntdll!RtlAllocateHeap+0xd7

0013f794 7c3416db 0000003f 7c3416f8 0000003f MSVCR71!_crtLCMapStringA+0x305

00000000 00000000 00000000 00000000 00000000 MSVCR71!_crtLCMapStringA+0x32d