viOpen hanging on Linux (close returns 0x3fff0082)

[danielbixby]

Hello,

I am running an application on Linux that uses NI Visa to talk to a 488.2-compliant device using a TCP/IP socket. Control and status of the device goes fine generally, but we've encountered some strange behavior for 2 error test cases.

Case 1: We unplug the Ethernet cable after having successfully opened a bus session with the equipment. The resulting sequence of events is generally as follows:

1. The code's next attempt to write data across the bus returns an error code as it should.

2. The ensuing call to viClose() returns a value of 0x3fff0082 (VI_WARN_NULL_OBJECT).

3. The subsequent call to viOpen() hangs, regardless of the time passed between the close and open.

Case 2: With the cable still unplugged, we restart the equipment model process which again tries to open a bus session.

1. The initial call to viOpen() returns 0xBFFF0011 (VI_ERROR_RSRC_NFOUND), as it should.

2. Same as Case 1, viClose() returns 0x3fff0082 (VI_WARN_NULL_OBJECT)

3. Same as Case 1, ensuing calls to viOpen() hang.

One would think that with the device disconnected from the network, the ResourceNotFound error would be returned for any and all calls to viOpen(). But the warning coming back from viClose() indicates that something strange is happening. Has anyone seen anything like this? 

Thanks

- Dan

NI Visa: 4.5.1

Linux: Redhat, 2.6.18-92.el5

[danielbixby]

A few more details:

I'm setting the attribute VI_ATTR_TCPIP_KEEPALIVE to true in my code, but his presence or absence hasn't made a difference. Someone suggested the Unix socket options SO_REUSEADDR, SO_REUSEPORT, and SO_LINGER as possibilities, but I don't see attributes in the NI Visa manual that correspond, at least obviously.

[Adam G]

Hi danielbxby,

I was wondering if I could get a little more information from you. I was wondering what GPIB device you are using and are you using a ethernet to GPIB converter? Also are you doing this in LabVIEW, CVI or which development environment? 

Regards,

Adam 

[danielbixby]

Hey Adam,

Thanks for your reply. The device itself is Ethernet, not GPIB. Specifically, we're using the C710-E10 controller from Universal Switching Corporation.

http://www.uswi.com/Products/CPU/C710.htm  

As opposed to the dev environments you listed,  we're talking to the equipment using our own application code which includes the NI Visa libraries.

NI Visa: 4.5.1

Linux: Redhat, 2.6.18-92.el5

Compiler: g++

As mentioned before, the hang on viOpen() occurs after physically un-plugging the Ethernet cable (Case 1), or starting the process with the cable unplugged (Case 2). Side note: the process remains frozen after plugging it back in, and a restart of the process is required at that point.

If it would help, I can provide some sample code with the scenario.

- Dan

[Adam G]

Hey danielbixby,

I was wondering what value you have openTimeout set to, the fourth parameter, in your viOpen command? You can also set the I/O timeout by using viSetAttribute() with the attribute VI_ATTR_TMO_VALUE. I was also curious why you try to call another viOpen after an error is returned. Have you tried have the code break after the first error is thrown?

Best Regards,

Adam G 

[danielbixby]

 Hey Adam G,

We are letting the VI_ATTR_TMO_VALUE attribute go to its default, which based on the NI-VISA Programmer Reference Manual is 2000 msec.

Our intent, after the initial unplug and fail, is to have our code attempt to reconnect every X seconds (in this case I had it set to 10 seconds) until the bus session is reestablished. So what we would expect, leaving the cable unplugged, is that each call to viOpen would return a RSRC_NFOUND (or similar) error code. Instead, the hang occurs as described.

Hope that makes some sense. I'll try to provide a code snippet today or tomorrow.

- Dan

[danielbixby]

Adam G,

I generated some test code and tried several plug-pulling scenarios in the lab today in hopes of providing you with a nice black-and-white setup to go with. However, the viOpen hang was never reproduced when I used the test code. This would appear to point to our existing infrastructure as being the cause of the problem.

Without revealing proprietary information, I can say that we are attempting to use a reconnect timer which is bound to a static function. The controlling process basically monitors the system for the timeout and invokes the static function when one occurs. The static function sets a reconnect flag to TRUE, and in the main program loop the reconnect is attempted (with viOpen) when this change is detected. 

I've verified that the VI_ATTR_TMO_VALUE is set to the default value of 2000 msecs. In your experience, have you seen viOpen hang? If so, do you know what else might cause it (e.g. resource manager or session properties changing)?

Thanks again,

- Dan

[Adam G]

Hi danielbixby,

We haven't seen any other cases where this happens at. One thing I would suggest to try and narrow down the problem is to look at the test code you created were the error was not reproduced and compare it to your existing code and see what the differences are in the error checking. Sorry I couldn't be anymore helpful without taking a look at the problem code. Let me know if there is anything else we can do.

Best Regards,

Adam G

[danielbixby · Accepted solution]

After digging into our application code I found the root of the problem. The code implements the reconnect timer by registering with the OS for SIGALRM's every N seconds (see the time.h and signal.h C++ files). I'm not completely sure about the ins and outs, but somehow the processing of the SIGALRM seems to interfere with the call to Visa and causes the hang. I'm pretty sure Visa creates its own threads, but that didn't seem to help the coexistence.

In short, the process timer has to be unregistered before I make the viOpen call. After the call returns, you can re-register and continue as usual. Note: this applies not only to viOpen, but also to any other Visa functions that perform I/O. (Functions like getAttribute and setAttribute are simply properties of the session itself and aren't interfered with).

Thanks for the tips, hope this is helpful for folks that see this phenomenon in the future.