RT Target drops out and glitches (skips a 3 second long beat)

[bradh]

We have a PXI chassis with PXI-8106 controller in it.  The target and host application will run fine for long periods of time 20minutes to 3 hours without problem.  The application is pretty much data acquisition on the target and displaying it back on the host.   

Intermittently, the data will quit streaming on the host.  After about 3 seconds, it will pick up again.  The problem is the host sends some status information (basically setting on controls that were made by the operator) it is sending over shared variables back to the target.  The target for some reason looses these during this glitch and it causes interlock conditions that ruin the test.  Though the target and host apps never error.

I have used the Real-Time system manager and we are only using a steady 20% of the CPU and less than 15% of the memory on the target.  It is almost as if it is a network collision problem, as if the target and host or waiting to exchange shared variable information.  Using the RT error log, the only reported error comes up in the LabVIEW error log as:

 snip:

#AppName: PH_EXEC_SMP

LVRT.DLL load address 0x05FBD000

Any ideas?

[Smootastic]

Hi bradh,

On which side are you hosting the Shared Variables?

Can you replicate this by shrinking down your code?  In other words, can you create a VI that transmits the status info?  Hopefully, we'll see the same problem and something to work from.

What version of Real-Time is running on the 8106? 

[bradh]

Kyle,

Answers to your questions:

1) LV 8.5.1 RT

2) The shared variables are on the target side.  We do have the Network Variable Engine (1.4.1) and the Variable Client Support for LV RT (1.4.1) installed on the target. 

Couple of observations:  It acts as if my consumer producer loops are "waiting" on the shared variables.  Where I can, I am using RT FIFOs when transferring data.  But there is quite a bit of controls/indicator info that is passed between the target and host (to facilitate the no front panel on the target).  To minimize the number of shared variables, I am using custom controls of cluster bundled controls.  RT FIFOs do not support custom control types.  Therefore those variables are just newtwork published buffered instead of FIFO.

Can I ask a basic question, that I thought I understood:  for a network publsihed shared variable (buffered or non-buffered) a loop that reads it will not wait for it to be written to; if it has not be written to, it will just use the last value and move on.  Correct?

I will work getting an examplar VI set together.

Thanks!

[StevenA]

bradh,

Can you describe your RT vi in a little more detail?  You mentioned a producer consumer setup.  What parts of this are on the RT? 

Generally I would suggest an architecture that has a single vi dedicated to the communications between the RT and the Host.  This vi is then the buffer between the rest of the RT code and the host.  The only type of data transfer between the communications vi and the rest of the RT are RT FIFOs.  What I typically do is make a set of vi's that convert my cluster data to data that is compatable with RT fifos then convert back to cluster data after it is transfered by the RT fifo. 

Having network communication of any type in your core RT code (including network shared variables) is not a good idea.

steve

[bradh]

The RT VI has a producer AND consumer loop:

Producer:   

• Performs DAQ I/O also does alarm and shutdown monitoring (just a sec on where the setpoints come from).
• Sends Data to lower priority consumer loop via RT FIFOs
• Receives setpoints from consumer loop via Net Shared Variable from the Consumer Loop*
• Control Logic also creates alarm/shutdown "indicators" and sends it to the producer via Network Shared Variable*

Consumer: 

• Receives Data from Producer Loop and writes to files
• Receives setpoints from host via Network shared variable and sends on to Producer Loop via Network Shared Variables*
• Receives alarm/shutdown status indicators from the producer loop and sends it host via network shared variables*

The host only acts as a front panel receiving alarm/shutdown indications and sending setpoints and commands.

*Used Network Shared Variables because RT FIFO can not use custom controls.  20-30 setpoints and controls with 15-20 status indicators.

Agree, my issue is probably with the network shared variables.

Can you elaborate on: "What I typically do is make a set of vi's that convert my cluster data to data that is compatible with RT fifos then convert back to cluster data after it is transferred by the RT fifo."  How do you not end up with a large number of Shared Variables or Arrays that are hard to track their contents as you add new functionality?

[StevenA]

Ok, it sounds like your producer loop is the core functionality of your control system.  You do have however some network shared variables in this loop.  The most simple change you could make would be to change all the variables in the Producer loop to single process (no RT FIFO, no network) and pass the data to and from the consumer loop with these variables.  Then in the consumer loop create network shared variables to pass the data to and from the host.  This is the most simple change for you to make because the single process variables can still pass the custom control data. 

This may solve your problem, it is not however the most ideal solution because the single process variables are shared resources between the producer and consumer loop.  Now, depending on the requirements of your system (i.e. speed and determinism) this may or may not be a problem.  It will however be much better than having the network shared variable in your producer loop. 

 If you don't want to have these shared resouces, then you need to implement an RT FIFO.  As you stated, you can't pass custom controls through an RT FIFO, but you can pass numbers and arrays of numbers.  So to make it more simple to manage these arrays and numbers create wrappers for your RT FIFOs that take your clusters and convert them to arrays or numbers, pass them throught the RT FIFO and then back to your clusters. 

[StevenA]

Below is an example of how you might do the wrapper for your RT FIFOs.  you do have to pass the data as arrays and numbers, however if you use a type def as illustrated below, you can still manage your data in clusters in your vi.  When you have to add data items, you just need to go and make the modifications to your type defs and make any appropriate changes to the wrapper vis.  It is convienent to keep your clusters grouped by data type.

Message Edited by StevenA on 10-06-2009 08:35 AM