Hard Crash of PXI controller with "plug-in" VIs

[CoastalMaineBird]

LabVIEW 8.2.1  WinVista, RT module.

Extra credit for this one:

  My app uses "plug-in" VIs.  That is, the user puts VI files in a particular folder, and configures a channel to use them.  They execute on each sample (10 Hz) and take measured data as inputs, producing some calculated output.  For example, one of them takes SPEED and TORQUE as inputs and produces HORSEPOWER as an output.  You then have HORSEPOWER as another channel: you can monitor it, graph it, set alarms on it, just like any other channel.

  The plug-ins reside on the host and are sent to the PXI box when needed (a hash/checksum scheme is used to detect changes and trigger a download).  There are no references to these VIs in the code, everything is done via the OPEN VI REFERENCE and CALL BY REFERENCE functions.

  A checking scheme is in place that makes sure that a plug-in's prerequisite channels exist before you're allowed to run.  It also checks the case where a plugin depends on another plugin, and ensures they execute in the correct order. 

  The checking scheme is performed on the host before a run, and again on the PXI when it gets configured.  The PXI requests a download, and stores the VI file if needed.

  ALL THAT HAS BEEN WORKING PERFECTLY for quite some time.

  I have a file called GLOBAL PARAMS.vi with 100+ constants on it (in tabs, so the UI is reasonable).  These are "constants" in the sense that they don't get changed during execution.  The programmer can change the default value to affect behavior.

  Some of the plug-in VIs use an item or two from the GLOBAL PARAMS vi. 

  Yesterday, I made a change to the GLOBAL PARAMS vi, namely removing one or two items that were no longer needed.  These items had nothing to do with plug-ins or the plugin process (that is confirmed).  No plug-ins used the items that I removed (that is confirmed).

  However, that caused the plugins to need recompiling (as I found out later).  Now officially, they needed recompiling before that, since they were last saved on LV 8.2.0, and I'm on 8.2.1 for a couple of months now.  But they worked just fine all the same.  I don't routinely open them, they're just there and they work, so I don't notice.

   But after removing those GLOBAL PARAMS items, I got a hard crash of the PXI box.  I mean hard. I mean it lost connection with the host, couldn't be contacted, and had to be rebooted.  Try again, same thing, same place.  I think it rebooted itself, because it came back after a couple of minutes.  More tries, exact same thing every time.

  Of course looking for session errors was no help, since it crashed and rebooted.  My own logging didn't work, since it was RAM based and was lost in the crash.

  In debugging it, I found that the place where I run the plugin to get its prerequisites was failing on the second VI it was checking.  The first one ran wiithout a hitch, but the second stopped at the CALL BY REFERENCE NODE for this particular VI. It would just hang there and never come back. I put this VI first in the list and it crashed on the same VI, so it wasn't because it was second.

  The VI itself could  be run on the host - it was just a quick calculation.  But every time on the PXI, it would hang and crash.

  I finally figured out that if I recompiled that particular VI on the host, it would work, but then it crashed on a different VI.  When I recompiled them ALL, everything worked perfectly.

  So, my guess is that the compiled code in the plugin referenced some particular offset into the GLOBAL PARAMS vi, but that offset had changed and the plug-in didn't know about it.  So it accessed some illegal space and crash-ola.

So what can I do about it?   How can I prevent this from happening again?

The check on the host went through the same process and didn't complain, presumably because it re-compiled the plug-in every time.  But it never saved the file.  If it had been saved on the host, then my hash scheme would have detected the change and downloaded the changed version and all would be well.  I don't know how to send a VI from RAM, only from a file.

I could save the VI file every time I check it, but that offends my sense of efficiency.

Hmmm.  I just now noticed the PROPERTY NODE -  VI - MODIFICATIONS - VI MODS property.  Maybe I could use that as a signal that I should re-save the file, before downloading.

Does that sound like an answer?

Anybody been here before? 

[falkpl]

I have seen some vi changes chich dont preak the code (i.e the run arrow is not broken) but the Labview Crashes when it runs and when the vi re-opens now the compiler has picked up that the code is broken.  I would assume this is happening since the plugin is referencing a changed VI (your globals definition vi) and hangs looking for the globals with a matching set of global controls.  Why are you using Globals for constants?  I admit I do this too sometimes, but for constants I like to represent them as a vi which has an indicator wired to a block diagram constant.  Now each constant is linked to a single non-changable value.  I have also implemented a semi constant which uses the constant unless the default apps config file has an over ride tag (essentially the constants name and a value to use as the override).  This method works for me and is very easy to manage.  This is not too bad of a code change, one vi replaces each indicator from the globals page.  If you are not using true constants (ie values set within the application) then ignore this suggestion.  I also usually make a constant "tree", a single vi with all the constants on the block diagram and an indicator fro each on the front pannel, so a single tree can tell me what all my constants are.

[CoastalMaineBird]

Why are you using Globals for constants?  I admit I do this too sometimes, but for constants I like to represent them as a vi which has an indicator wired to a block diagram constant.  Now each constant is linked to a single non-changable value.

Well, in general, using a global seems like less memory space.  Also, I change some of the "constants" if I am in development mode. For example, the client's default settling time is 60 seconds, so I set the default value to 60.  But I detect that I am running on my development machine at startup and change that constant to 2 second so I don't have to wait so long.

I don't see, though, that by using your method, I would avoid the problem. If I removed a constant, I would save the CONSTANTS vi, but the plug-ins still wouldn't notice it.  When I do the checking on the host, it would re-compile the plugins, but the file itself would remain unchanged.  So the copy on the PXI is outdated.

I have to either transmit the VI from RAM in the host, or detect the changes and SAVE the file before transmission. 

Message Edited by CoastalMaineBird on 10-22-2008 08:59 AM

[falkpl]

I was not suggesting a single VI called CONSTANTS but a single vi for each constant.  This usually allows me to decouple the constants from each other, so changing one constant will only effect vis which directly call that constant.  Note that constants are vis so they can automatically change their value programatically ie (in exe use 2 seconds otherwise in development use 2 seconds).  I agree that this is not the most memory efficient method of calling a constant but is loosly coulbed and easy to maintain (add/remove constants have minimal impact on code).  For RT systems a "functional constant" might not make sense. 

[CoastalMaineBird]

I was not suggesting a single VI called CONSTANTS but a single vi for each constant. 

I see - I was misunderstanding you.

I believe that would avoid the problem, indeed.  If I decided that this constant is no longer needed, then I kill the VI, and be done. It doesn't affect anything else.

That's a large price to pay, though.  I have 100+ constants in my GLOBAL PARAMS.vi.  My client will have programmers, and I wanted them to see all the programmer-adjustable things at once, or at least with a tab click or ten.  Having 100+ VIs complicates that.  In addition, the time penalty (for a VI call) is not insignificant.  Given that I have 30+ plugins, each running every 0.1 second, I need to stay conscious of that.

[LaRisa_M]

I don't think you'll have to worry about preventing this from happening again. It's always suggested that one do a mass compile when updating LabVIEW versions, especially in a large project with many global variables and sub-vi's. It sounds like you got away with it for awhile before the problem started. Are you still having to compile the VI's after any changes in constants before you deploy, or is everything working business as usual after the big mass compile?

[CoastalMaineBird]

 It's always suggested that one do a mass compile when updating LabVIEW versions, especially in a large project with many global variables and sub-vi's.

But it's not the LV version change that killed me, though. I had been working fine for months with LV 8.2.1, and these plug-ins were sitting at 8.2.0.  I don't load them into LabVIEW myself normally, so I didn't even notice.  There wasn't any significant difference involved, so they worked OK.

But when I removed the global constants, the host re-compiled when it loaded the plugins (when I was checking them).  But I didn't know it.  But when the PXI loaded them, it needed to recompile, but there is no compiler on board.  So apparently, it went with the code that it had.  And the code that it had referred to a batch of globals that had been rearranged since the code was compiled.  So ka-blooie.

When the PXI code opens a plugin VI reference, it checks the VI execution state via a property node.  If that returns a state besides "Running" or "Ready", then my code will complain to the host.

Hmmm. maybe there's my problem.  When getting ready to run a plugin, I perform an OPEN VI REFERENCE, then a GET EXECUTION STATE, then I RUN it (to get the prerequisite channels), then I close the reference (I'm doing a dry run first).

I pass the error through those items, but I don't pay attention to the EXEC state until AFTER I've run it.  If all that shows no error, THEN I check the EXEC state, and use that to decide if I can run.

Perhaps it was telling me that it wasn't ready to run,  but I didn't see that until AFTER it runs and is closed.  I suppose it's not an error to load a VI that cannot run.

So I should change my logic to check the EXEC state BEFORE I try to run it.  D'oh!

I'll fetch an old plugin from a backup tomorrow and try it again, watching what the EXEC state returns to me.

[CoastalMaineBird]

That wasn't the problem.  

I moved one of the older plugin VIs (that needed recompiling) back onto the PXI box.

Remember my sequence is:

Get path - Open VI Reference - Get EXEC STATE - CALL BY REFERENCE, CLOSE, Check for Error, Check Exec State. 

I put a breakpoint  at the OPEN REFERENCE and followed the logic.

OPEN VI REF returns no error.

Get EXEC STATE returns no error and state = "Running" 

(I happen to think that's the wrong result, since it's not really running.  It used to return "Idle" in LV6,  but it's been this way since LV 7, so I deal with it. )

It does NOT return BAD. 

Then when I CALL BY REFERENCE - ka-blooie. 

So checking the EXEC state first would not avoid the problem.

 The problem is that the PXI will go ahead and execute a VI with bad code.

If I SAVE the plugin file on the host, then my change-detection logic works, it sends the new version over to the PXI, and all is well.

It looks like my answer has to be looking at the VI MODIFICATIONS flag, and SAVING the file if it's changed. 

[CoastalMaineBird]

OK, I fixed it so this won't happen again.

When I check the plugins on the HOST, I now pay attention to the MODIFICATIONS flag for each VI.  If that is nonzero, I call the SAVE method.  So the file on the HOST gets updated, and that changes its hash value.  When the hash values mismatch, my code automatically downloads the new version to the PXI, so we're all set.

 Are you still having to compile the VI's after any changes in constants before you deploy?

For the record, it wasn't a change in constants VALUES, it was the fact that some were REMOVED from the globals VI. 

[LaRisa_M]

That's some snazzy programming!