FP-DI-301: All channels went "HIGH" at the same time.

[chiphowell]

Will FP-1001 "Watchdog" settings or FP-DI-301 module/channel settings cause digital inputs to go "High" regardless of the connected swithes positions?

On 4/29/2009 @ 07:11:30 AM, while running a known good process, all of my FP-DI-301 input channels (48) on a individual network bank went "High"!!!

I am currently running a "Lookout" process with 6, FP-1001 network banks connected via rs485.  This process has been in operation for more than five years.  On the date in question, maintenance personnel informed me that a machine had gotten jammed and they had cleared the jam and it was now running.  After watching the video playback of the machines operation, I realized that it did not operate as programmed.  I also noticed that three other like machines associated with the same network bank performed EXACTLY the same way at EXACTLY the same time.  All other machines associated with the other 5 network banks functioned normally.

After reviewing the collected data (citadel 4 database) I found that all of the channels of the DI-301's went "High" together for a period of time, then they reverted to normal operation for a few seconds and then go "High" again in unison.  This back-and-forth continued for aproximately 15 minutes, until the whole process was shutdown and restarted.

All of the channels are connected to field mounted prox switches, there is no way that all the switches toggled/sensed at the same time.  This has to be a network/module problem. 

[Smootastic]

Hi chiphowell,

Assuming that you have enabled the Watchdog timer and configured the output value for each channel on the DI-301, then the inputs will read High.  Since the process has been restarted, have you run into the same problem?  Has anything changed in your setup recently? (I know you mentioned this has been running for 5 years).

[steveward]

Just to update you.  The same problem has happend again.  Right now Chip is at the plant.  Lookout is showing the inputs as blinking on and off, but the LEDs on the field point modules are not changing.  This issue is only happening with this cabinet of field point modules.  There are other banks in the chain that appear to still be fine and have stable inputs.

Any ideas?  Are there any bios or driver software issues that could cause this?

Steve

[Smootastic]

Hi Steve,

What version of the software are you using?  Do you have extra FP-1001 modules to see if this is caused by the network module?  Does replacing the DI-301 module fix this issue?  We need to determine if this is related to the software or hardware.  Perhaps an upgrade of the drivers may help, but I would test the hardware if you can.

[chiphowell]

Kyle,

This is Chip I am back from the field.  The current "Lookout" version is 4.0.1 (build 51).  I double checked the "Fieldpoint" configuration from the Fieldpoint software then opened Lookout, selected my fieldpoint object and re-imported my .iak file.  I have not been able to re-create the incident that would cause my switch inputs to blink ON then OFF when viewing the logical display but remain steady ON at the actual module channel.

All my problems are associated with address banks 20 & 30 only(mounted in same j-box), address banks 00-10 & 40-60 are all performing normally.  These two banks are also the newest modules purchased approximately 4 yrs. later than the others, could there by firmware compatibility issues? 

Some more Problems and/or Symptons I have discovered.

1.  RED X's on control panel, associated with address bank '30':  After noticing the x's I went to the module bank and discovered that my power led was lit but, the ready led was not lit.  I pushed on module in position #1 and waalaa my ready led lights up.

• Replaced FP1001 with another I had in storage. (Not sure why I had an extra, this one could have been faulty also not sure):  Problem still occurs.
• Replaced IO module in position #1:  Problem still occurs.
• Replaced Terminal Base in Pos. #1:  Problem still occurs.
• Replaced Terminal Base in Pos. #2:  Problem still occurs.
• I noticed that pushing on the modules in positions 2-5 had no effect on the ready led.  I could only get it to come on after pushing on module #1.  I replaced the FP1001 again, this time I used a known good one from address bank '10'.  Problem has not reoccured since.

2.  Outputs de-energize for a period of time 5-20 seconds then come on again.

• This is a weird one also.  I have 4 machines controlled be a 1 sequencer object each.  During the sequence operations all 4 machines will just stop like they are catching there breaths.  After about a 5-20 second rest they will just start back up and continue from where they left off and finish there sequence as programmed.  I have not been able to troubleshoot this problem yet, but I am pretty sure all my outputs are going low then returning to there programmed states. 

Headed back June 8th - 11th.  Will only be able to monitor for responses and update conditions after hours when back at hotel.

Chip.

[chiphowell]

UPDATE TO PREVIOUS REPLY.

Another proble/symptom that I forgot to mention, may be helpful.

• The process file (L4P) that is currently running has a corrupt LKS file, unable to recompile:  If possible, how do I find the bad code and repair it. 

Chip.

[Smootastic]

Hi Chip,

I'm not too familiar with Lookout, but in the event this may be software related, posting this information in the Lookout Forum Board will make sure that we can also get some help from those more familiar with that product.

As far as the hardware is concerned, I'll take a closer look into seeing if firmware compatibility may be an issue.  Typically, we recommend upgrading to a newer version of the firmware, but it's interesting that the older FieldPoints are working correctly.

[chiphowell]

Kyle,

Found the problem with the LKS file, repaired it.  It seems that I enclosed a sequencer object: state-label in quotatiion marks.  The LKS file also encloses the label in quotes and kept on enclosing it in quotes.  Must have been a couple thousand quotation marks in twenty lines of code.

I don't know, but maybe this has something to do with my input problem.  Those corrupt lines of code were part of the sequencer object that controls the machines that are acting up.

Will have to run the process tomorrow to be sure.

Chip

[chiphowell]

Kyle

A lot have things have happened and have changed.

Original problem #1:  Lookout 4.0.1 sees my inputs (All 48, DI-301's)@ Fieldpoint address bank "30" turning on and off, staying on, changing state, etc.... for a short period of time.  This problem happened once at the end of April (all came on and stayed on for about 15 minutes) and again about mid-end of May (This time they were blinking on and off).  During the last event I noticed that the actual DI-301 channels were stable and were not changing state like Lookout was seeing.

I think I have fixed this problem.  It hasn't shown up again and I cannot recreate it.

What I have done to address the above problem:

• Cleaned out the PC (NT os), uninstalled unneeded programs.
• Reconfigured all fieldpoint 1001 watchdogs.
• Imported my reconfigured IAK file into my fieldpoint object of my Lookout process.  

Problem #2:  While troubleshooting problem #1; I found, that occasionally upon power-up I would not get my "ready" LED's on my I/O modules of address bank "30" and subsequently red X's would appear on some of my Lookout panels. 

What I have done to address the above problem:

• If you push on the first I/O module the ready lights will come on.
• I replaced the FP1001 and other modules and terminal bases and this problem would not go away.  I swapped out the FP1001 again, with a known working one from address bank "10", and problem dissapears.  Strange that it does not show up @ address bank "10" now.

Current problem #3:  Programmed machines associated with address bank "20" (1-FP1001 and 5-RLY420's) will stop for about 10-30 seconds then restart and finish their program.  This may only happen once every 2 days or 3-4 times in one day.  Occassionally I will get a fieldpoint generated alarm "Channel Specific error, DEV:21-24, CH: 00-07, ERR:3"  I even got "Channel Specific error, DEV:22, CH:09, ERR:3".  The RLY-420 module @ this address does not have a channel "9".

What I have done to address the above problem:

• This is when I found and repaired the corrupt source code, LKS file.  Problem returned the next day though.
• I have found out that the outputs are de-energizing.  That's all so-far.

Could changing baud rates or poll rates and times help?  Could some sort of weird comm interference cause my outputs to de-energize and not give me a COMM fail?  I am still thinking it is fieldpoint related somehow since the LKS file fix did not help. 

Chip.

[Smootastic]

Hi Chip,

What version of the FieldPoint driver are you using?  Maybe upgrading versions will help.  Also, I don't quite understand what you mean by "de-energizing."  Can you elaborate?  Thanks and hopefully we can get you back up and running.