GPIB-ENET lock up, solaris 9

[steveshirley]

Good day.

We are experiencing lock-ups of our GPIB/ENET boxes requiring power cycle.  The lock-ups

are occuring with a Solaris 9 host.  In some cases, the lock-ups appear most commonly

following a security scan of our Unix systems (and perhaps these GPIB boxes as well).  They

also occur far less frequently when no such scan has taken place.

The lock ups occur when no activity is present and often after long periods of

inactivity (but with the interface live) [long being defined as hours or over a weekend].

We're capturing things in the debugger at present and I don't see anything on the boards

that directly addresses our problem.

Pointers would be greatly appreciated.

GPIB-ENET/100 boxes, Part 186852G-01 Rev 1.1

Thank you.

[Shelby S]

Hi,

I just want to ask a couple of questions to get a better feel for what's going on.  When the GPIB-ENET locks up, is there any communcations going to the device at the time, or is it completely inactive?  Also, what type of activity is being displayed on the lights on the front of the device?  After a power cycle, does the device lock up again without attemting to communicate with it?  And what version of our drivers are you using?  Thanks!

[steveshirley]

As for the lights, here are desciptions of what we normally see and what we see during "hangs".  The

"core" refers to our GPIB process that feeds other Unix processes.

                /----------------  eNet Box Lights --------------\

                       ----- ENET ----------       ------ GPIB ----
         GPIB          Link
Core     Comm   Pwr    10/100   Tx       Rx        Talk     Listen
------------------------------------------------------------------
Running  None  Yellow  Green  Blinking  Blinking   Off      Green 
                              Red       Red 

Not      None  Yellow  Green  Off       Blinking   Off      Off   
Running                                 Red


Running  Comm  Yellow  Green  Blinking  Blinking  Blinking  Blinking
                              Red       Red       Green     Green 
                                                Mostly On   Mostly On


The first row shows the states of the lights for both immediately after 
the core is started and after there has been some comm and then the comm stopped. 

The second row was observed after a functioning core was killed.  

This is what Ken and I both observed following separate eNet box hangs:

Here is the status of the lights on the ENET/100 box attached to su57-1. 
Pwr/Rdy - on (solid yellow)

The 3 lights with label "ENET" are as follows:
UNK 10/100 - on (solid green)
TX  - off
RX - on (pulsing orange)

The 2 lights with label "GPIB" are as follows:
Talk - off
Listen - off

The lights that Ken calls "orange", I call "red".    

We've also tried poking around with ibic.  Again, core and CDS refer to our applications - CORE being
attached to the GPIB and CDS being the user interface.


NOTES 

[As previously noted my macro doing memreads on su57-0 hung at around 19:07 last night.] 

POKING ON GPIB0 

Initial state of the su57-0 eNet box lights: 
   Pwr: Yel;  Link: Green; ENET RX: Blinking Red 

Ran "ibic" and it came up okay.  (After the fact I saw that "ibic" is pulling 
the command from /bin.) 

Tried "ibfind gpib0" and it hung.  I waited over two minutes and then gave up.  I 
had to kill ibic with a kill command from another window.   

Tried "/usr/bin/ibic" it came up OK. 

I tried "ibfind gpib0" and it hung again.   

I stopped ibic with a kill command.  I killed the core for su57-0.  [I thought.  I actually took out the su57-1 core at this point.]   

I started /usr/bin/ibic again and tried "ibfind gpib0" again.  It hung.   

I killed /usr/bin/ibic.   

I power cycled the su57-0 eNet box.  I saw the eNet boxes do some blinking 
following the turn on that I had never noticed.  The lights ended up in this 
state: 
   Pwr: Yel;  Link: Green; ENET RX: Blinking Red 

I note that the Pwr light blinks at a fast, but constant rate for several 
seconds after power on.  I note that the Link light is red for several seconds 
following power on.  I also saw a couple of green strobes of the Talk and 
Listen lights.   

I started /usr/bin/ibic.   

I performed a "ibfind gpib0" command and got back 
"id=16384".   

I entered "ibsic" and got back: 
[0130]   ( cmpl cic atn ) 

In another window, I tried to start a mmc_core on GPIB0.  The start_core 
failed.  [Because the core is still running because I killed the other core earlier.] 

I performed the ibsic command again and got the same 0130 response.   

I quit out of /usr/bin/ibic. 

I discovered that I had previously killed the wrong core.   

I killed the correct GPIB0 core. 

I ran /usr/bin/ibic and did the "ibfind gpib0" and ibsic and got the same 
response as before.  I quit /usr/bin/ibic.   

I started a new mmc_core on GPIB0: 

hardware  7779  1  0 11:39:25 pts/17  0:00 /apps/cds/cds_910t6/bin/mmc_core 

I started CDS on GPIB0 and the Listen light came on Green on the eNet box.  I 
exited CDS and the Listen light remained on. 

I started /usr/bin/ibic and entered "ibfind gpib1" and it hung.  That seems to 
confirm that the eNet box is in such a state that it hangs ibic with and 
without a core running.   

I turned off GPIB0 eNet box. 

I started /usr/bin/ibic and then entered "ibfind gpib0" and it seemed to 
hang.  I waited a minute or two and got back:   
  id = 16384 
  [8100]   ( err cmpl ) 
  error: ENEB 

I did an ibsic and immediately got back: 
  [8100]   ( err cmpl ) 
  error: ENEB 

I exited /usr/bin/ibic. 

I started /usr/bin/ibic again. 

I did "ibfind gpib1".  The gpib1 eNet box has not been power cycled.  I waited 
6 minutes and it never came back.   

I got out of /usr/bin/ibic and the started it again.  I did a "ibfind gpib0" 
and timed how long it took the [8100] status to come back.  It took around 3 
minutes and 47 seconds. 

I tuned on the eNet box and tried to start CDS in another window.  It seem to 
hang.   

I exited /usr/bin/ibic.   

The CDS window came back with: 
 *E* IEEE-488 Error: GPIB initialization error. Please contact CDS support. 
 Error occurred while updating cluster status 

When I would exit and start CDS again I got the same message.   

I killed and restarted the core and was then able to get into CDS.  I exited 
CDS.   

I ran /usr/bin/ibic.  I ran "ibfind gpib0" and noticed that it causes the 
Listen light to go out.   

I started CDS and it came up OK.  The Listen light came on.   

I entered "ibsic" in the ibic session and got back "[0120]   ( cmpl cic )" 
instead of the 130 status.  The Listen light went out.   

I did a cst command in the CDS window and the Listen light came on green and 
the command functioned normally.   

[Christopher_T.]

Hi steveshirley,

Just to verify, when your device hangs, you don't see anything strange with the power light. It is just a steady yellow (does it blink at all?) Do you happen to know the firmware

of your GPIB-ENET/100?

Thanks!

[steveshirley]

Hi Chris.

Well, the lion's share of hangs occur while we're not actually doing anything much

with the boxes so we are not paying attention to the lights (that is, there isn't

anyone in the lab when they hang, most of the time).

As for the firmware, we are running B.9 according to EthernetConfig.  The

boxes in question are Part # 186852G-01 Rev 1.1 SN D1E122 and D18B7D.

We would like to know if we're running the latest 'driver' sw and I don't see anything in

pkginfo or prtconf or modinfo that gives me a clue as to the 'driver' version if there is

such a thing for these boxes - how do we know that we've got the latest and

greatest?

The readme.txt in the install directory says release notes NI-488.2M 2/2001 V 1.4 [version of the

release notes?  of the sw?] if that helps.

Thanks!

[Shelby S]

Hi,

It appears this is an older version of the firmware.  Do you have Measurement & Automation Explorer installed on your system?  This would show us the driver versions and other useful information.  I'm not sure if this is installed with our drivers on Solaris, as it is for Windows, but take a look.  Thanks.

[steveshirley]

Thanks.

I don't see NIMAX or anything like that installed on our systems or in the TAR that came with

the box.  Is there some encoding in gpibrc that would tell us the version?  (The C library is

dated 2001 ...)

[Christopher_T.]

Hi steveshirley,

There should be a readme with the driver. I am currently looking into the issue and see what next steps would be available.

Thanks!

[wsmills]

We are having the same problem with one of our customers. We can force a lock up with a security scan of the ip address the gpib is located. Our version firmware showing on m&a explorer is 4.2.1.3001. Was wondering if there are any updates we could apply to remedy this issue.

[Christopher_T.]

Hi all,

So there is a firmware update for the GPIB-ENET that did help with stability issues. It might help out the issues you are seeing.

Here is the link with more information:

http://digital.ni.com/public.nsf/allkb/F4F7F2F74A60819A862569DD0071180F

Hope this helps!