Shared variables on cFP become zero while communicating with cFP

[Underscore_c]

I am running a PC with two network cards, one on a wired ethernet, the other on a private IP directly connected to a cFP-2120 (running full 6.0.5). I am using shared variables on the cFP and aliasing them to shared variables on the PC (originally running LV 8.6, now 2009f2 with same problem in both). The PC variables are being logged to a Citadel database. I have the cFP program built to run at startup, and it runs fine when not connected to the PC.

When connected to the PC all of the shared variables on the cFP will occasionally (almost 1/min for several seconds) and randomly become zero. Even if they are scaled not to allow becoming zero. I can tell this is happening because I am using shared variables as relay module alarm setpoints, and the relays will suddenly click on. I confirmed this by checking what the cFP writes to its internal flash card, and indeed the variables are suddenly all zero. On the PC it says it cannot communicate with the shared variables. This happens even if LabView is not running on the PC!  In the Distributed System Manager shared variables on the cFP are listed as (Disconnected) during that time, but it continues to be able to read the correct live values of the cFP module inputs.

Interestingly, I have an identical computer with the identical setup (and a different cFP-2120) that does not have this problem. Both PCs are new from the factory. I have tried disabling the second ethernet adapter, shutting down all firewalls/virusscan, and re-ordering the network adapters in the Windows networking/Advanced settings without success. Simply unplugging the ethernet cord between the cFP and PC does not cause this problem, and the shared variables becoming zero does not trigger the network watchdog settings on the cFP.

My questions are:

1) Can this problem be fixed?

2) Losing communication for a few seconds is OK, but having the cFP variables become zero is not. Can I prevent the cFP variables from becoming zero?

  --David

[DirkW]

HI David,

I guess both of the controllers have the same software installed? Could you try the second controller with the PC that normaly fails?

Has your controller a DNS address assigned that is valid. If you are not sure about it just disable it to 0.0.0.0.

Let me know.

DirkW

[MarkBlack]

Can you clarify a couple of points?

•  The SVs experiencing the issue are hosted on the FieldPoint and there are SVs hosted on the PC that are bound to the cFP's Shared Variables in order to log to a Citadel DB.
• "In the Distributed System Manager shared variables on the cFP are listed as (Disconnected) during that time, but it continues to be able to read the correct live values of the cFP module inputs" - therefore all of your SVs have a value of " 0 (Disconnected) but all of the other modules are returning the values read at their inputs.
• Are the SVs experiencing the issue writing or reading data?
• Is there an error log produced by the cFP during this time.   You can access these logs in MAX by right-clicking on your cFP in Remote Systems and selecting View Error Log.  Each time your cFP boots a new log file is created.

 "Losing communication for a few seconds is OK, but having the cFP variables become zero is not. Can I prevent the cFP variables from becoming zero?"  

 

Since it sounds like you have LabVIEW DSC this can be done.  In the property page there is a category for initial value.  You can set this value to be whatever you desire and it becomes the new default value from the Shared Variable, which is usually zero.  

[Underscore_c]

DirkW,

Both controllers had the same software when this issue started (whatever came with LV8.6). I've since reformatted and upgraded the problematic controller with the OS that came with LV2009, but the problem remains. Unfortunately I can't take the other controller out of service, but if I can get another 2120 backplane at some point I will try switching them and post it here.

There is no DNS server set on either the PC or cFP. I set the PC to be IP and gateway 192.168.1.1, and the cFP to be 192.168.1.2 with the PC's gateway, subnet mask on both as 255.255.255.0

  --David

[Underscore_c]

Mark,

"The SVs experiencing the issue are hosted on the FieldPoint and there are SVs hosted on the PC that are bound to the cFP's Shared Variables in order to log to a Citadel DB."

Yes, that is correct. You gave me an idea and I tried unbinding one of the variables on the PC. But it still goes to zero with the rest of them on the cFP.

"In the Distributed System Manager shared variables on the cFP are listed as (Disconnected) during that time, but it continues to be able to read the correct live values of the cFP module inputs" - therefore all of your SVs have a value of " 0 (Disconnected) but all of the other modules are returning the values read at their inputs."

Yes. Actually they either show their last good value grayed out next to (Disconnected), or the values just freeze. But you are correct that it still shows the raw module inputs updating during that time.

"Are the SVs experiencing the issue writing or reading data?"

No, as far as I can tell they are being written correctly on the cFP, read correctly by the PC, and recorded to the cFP's drive and the PC's database properly. All are read-only except for a couple that also seem to be written to the cFP correctly.

"Is there an error log produced by the cFP during this time.   You can access these logs in MAX by right-clicking on your cFP in Remote Systems and selecting View Error Log.  Each time your cFP boots a new log file is created."

I see the error logs from each reboot, but they don't have anything else logged other than this (which maybe is normal and I assume occurs during the boot-up):

#####Date: Thu, Oct 22, 2009 11:29:43 AM

#OSName: PharLap ETS NtStyle

#OSVers: 13.1

#OSBuild: 257

#AppName: PH_EXEC

#Version: 9.0 32-bit

#AppKind: AppLib

LVRT.DLL load address: 0x0040F000

"Since it sounds like you have LabVIEW DSC this can be done.  In the property page there is a category for initial value. You can set this value to be whatever you desire and it becomes the new default value from the Shared Variable, which is usually zero."

That was my first thought, and I tried setting both the initial value and the scaling coerced to a range above zero (on both cFP and PC shared variables for good measure). But sadly it seems to have no effect on going to zero (it does initialize/coerce correctly otherwise).

  --David

[MarkBlack]

David,

Its sounds more and more like you are starving our the thread the network variable engine is running on.  This would explain why you shared variables become disconnected but your I/O items are still updating.  Can you tell us a little more about what you are doing in your FieldPoint code (screenshots are always helpful)?  

[Underscore_c]

Mark,

I'm not sure whatwould be helpful to tell, but I'll try. Screens attached at bottom.

Basically I am reading Analog IN from two modules, Digital In from one module, and writing to a Digital Relay module. I write everything to, and generally read everything from, the shared variable nodes. At some defined interval I write the shared variables to the flash card on the cFP (here I've inserted a "False" constant so it never writes since when variable "Iteration to Save" goes to zero it goes to town writing). All errors are suppressed and passed out to be read by the PC.

There's not much processing as you can see in the screenshots. Originally I had a timed loop running at 500ms, but here I've changed it to a while loop executing every 500ms (thought it might help, it didn't). I've also set the cFP bank pause time to 10ms in the project properties so it is generally very low CPU and memory usage (about 25% each according to the DSM).

  --David

Message Edited by Underscore_c on 10-22-2009 04:52 PM

[MarkBlack]

Your code looks pretty good.  Nice layout!  I would suggest connecting the error terminals for your Shared Variables since this will allow LabVIEW to better determine execution order.  This will allow for your code to execute more efficiently.  Alliteratively you could use the Shared Variable API instead of the Shared Variable Node, but this probably would be more work.  

The file I/O operation is also probably pretty expensive; especially since you are using the express VI.   It will probably be more efficient if you open the file in the beginning of your program and write to it continuously until the program shuts down.  This may not be a trivial change depending on you application, and I would hold off making this change until we can determine if CPU usage is the cause of this behavior.  I suggest using the diagram disable structure to "comment out" parts of your code (such as the Write to Measurement File VI) to see if we can avoid the issue.  

Cheers,  

[Underscore_c]

Mark,

I had thought about the expense of the file VI, but eliminating it didn't help. I also wired up the error terminals; good to know that can improve performance.

I ended up solving my problem by re-installing the OS (XP) and then doing a fresh install of LV 2009f2. I'm using the same program on the cFP and PC, and the shared variables no longer drop out.

  --David

[Underscore_c · Accepted solution]

To update my earlier post, it turns out re-installing was not the solution (the problem of variables going to zero started occuring again). I finally realized that this only happens when I set the cFP Time Server IP (under Additional Configuration in MAX). I was setting to the Time Server IP to 192.168.1.1 (the IP of the adapter the cFP was directly connected to). However, clearing this setting fixes the problem.

I don't suppose the time on the cFP should need to be set too often (and perhaps it is syncronized if I ever re-deploy files to the cFP?), so I am assuming leaving it blank is fine. Oddly, I have the same Time Server IP setup on another cFP and PC (running LV 8.6) without problems.

  --David