NI Serial--Unknown System Error

[BenjaminAZ]

All,

We have having a bizarre issue with NI-Serial.

We have a Windows XP SP3 system with NI Visa 4.5.1, NI Serial 3.4, and NI MAX 4.6

Our standard PC configuration has 7 motherboard COM ports, 4 ENET-232/4 converters, and 1 ENET-485/4. We've built 6 stations with this configuration and they all work without any issue.

We've built a duplicate system, but it will intermittently have error BFFF0000h (Unknown System Error (Miscellaneous system error)) when issuing a ViOpen call to various COM ports (NI-Serial only ports, not the motherboard ports). When the error occurs, it will go hard over. The only way to correct it is to shut down the pc (rebooting doesn't even clear it). After some time, the issue will occur again. Yes, ViClose calls are always issued.

We don't believe it's the software setup or NI software versions: we have gone as far as applying a Norton ghost image of a working system (so...working XP, Ni-Serial, NI-Visa)...but the error will still occur on this one station. All of our other systems use Visa 4.5.1 and Serial 3.4....

We've characterized this problem and have come up with a way to provoke the failure reliably. Any NI input on this would be helpful.

The following sequence of events will cause the failure:

Assume the following COM port assignments:

device 1: ENET-485/4---COM 8 - 11

device 2: ENET-232/4---COM 12 - 15

device 3: ENET-232/4---COM 16 - 19

device 4: ENET-232/4---COM 20 - 23

device 5: ENET-232/4---COM 24 - 27

1) power is applied to all 5 serial ENET devices.

2) After all units have finished booting, issue a viopen call to COM 20 (device 4) and 24 (device 5). SUCCESS.

3) Issue viclose to COM20 and 24

4) Shut off power to devices 1 - 3. 

5) issue viopen call to COM 20 (device 4) and 24 (device 5) again. SUCCESS

6) issue viclose to COM20 and 24

6) Log off.

😎 Log back in with the same account

9) issue viopen call to COM 20 (device 4) and 24 (device 5) again. FAILURE: BFFF0000h (Unknown System Error (Miscellaneous system error))

At this point, the failure will now be hard over. I've tried the following, none of which will clear the error state:

1) restore power to devices 1-3

2) log out, log back in

3) reset power to network switches

4) issue NiSerE_U –stop and –start

5) reset the PC.

6) resetting power on devices 4 and 5.

7) logging in as root

In addition, attempting to open these ports via hyperterminal will also report a system problem with the port.

ANY help with this would be appreciated.

We went as far as replacing the PC chassis and C drive with a known working PC, but once installed, the problem showed up. It apparently is not the PC hardware or SW...

Here is an example of our system's network topography:

PC <---RJ45---> Cisco 3560 switch 1 <----fiber---> Cisco 3560 switch 2 <----RJ45---->  Cisco 3560 switch 3 <----RJ45---->  Cisco 3560 switch 4 <----RJ45----> ENET-232/4 1 and 2

[dstrandb]

Hi Benjamin,

Thanks for providing such a detailed description of the issue you're running into. I just want to clarify one thing. When you say that the system goes "hard over" what do you mean exactly? Is the only error you get the BFFF0000h? 

The issue may be due to the way that Windows is trying to name the ports. I might suggest setting the port aliases to some other names that would have no potential to conflict with what Windows might name the motherboard ports, and then see if the error persists.

[BenjaminAZ · Accepted solution]

Thanks for replying dstrandb 🙂 After fighting this issue for the past 3 months, we have finally found the solution yesterday and confirmed it with testing this morning.

Our windows XP machine is a client of a PC with windows server 2003. The Server PC has software installed called "DeviceLock" which controls which groups can access the DVD drive, USB, etc... This software apparently does not work with NI Serial. We used ProcessMonitor to determine that while starting the NiSerEU service manually, DeviceLock was being invoked--after this occurs, it is no longer possible to open NI serial COM ports, even with hyperterminal

we had to restore a ghost before the client was joined to the domain. Thorough testing revealed that this did fix the error. After joining to the domain, we could no longer open NI Serial ports with hyperterminal or programatically with viOpen and viClose.

[dstrandb]

Hi Benjamin,

I'm glad to hear it! I haven't heard of DeviceLock before, but I'll check to see if issues between DeviceLock and NI Serial have been documented before, and if necessary I'll write something up, so that in the future, we can identify this issue sooner. Have a good day!

[BenjaminAZ]

Thanks, dstrandb, I'd appreciate it. This problem cost us a lot of time and money to solve.

We ran further tests and confirmed with 99.9999% certainty (can never be 100% sure right? )

We never had any problems communicating with motherboard COM ports with DeviceLock....I'm not sure which mechanism NI uses to fool the system into thinking that ENET ports are physical ports, but DeviceLock appears to trigger as soon as NiSerE_U starts up.

I'd be happy to provide our troubleshooting techniques or any data you need to make this more of a known issue...this way other customers don't have to go through such pain, heh.

[JasonSmith]

I don't think it has anything to do with how NI-Serial makes the ports appear to be locally attached serial ports, but everything to do with the network configuration. I would guess that Device Lock is configured to block all unknown network communication, which would include communication to the ENET serial device.

It looks like Device Lock can probably be configured with IP addresses which it allows traffic to. You will just need to configure an exception for the ENET serial device.

[BenjaminAZ]

I still find it strange that it will allow the traffic as long as all NI ENET converters are powered up...but as soon as you shut off some and re-log it goes hard over...

If DeviceLock was truly blocking unknown network traffic, it should block it regardless of whether the converters change state in power.

One thing we noticed with ProcessMonitor, is on a failing run, no TCP communication is issued to the converter. On a passing run, we saw a socket opened to 192.168.100.35:5225 with packets sent back and forth. On a failing run, NiSerE_U is trying to issue a "reconnect" tcp call to the converter that was turned off, rather than talking to the converters that are still powered on.