Program Stalls Briefly causing test rig damage

[ArcticCat]

I am using LabView and a C-DAQ 9178 to control a hydraulic test rig.  The C-DAq utilizes a USB connection to the host computer running Windows 7. The program is a closed loop setup utilizing a PID subroutine to control each of the hydraulic valves and pump.  The control can be based on either cylinder position or load.  I am using a 9219 module to read the load cells, position indicators, and valve spool position.  A 9264 module takes care of the voltage outputs to the valves, relays, etc.The program will run great for extended periods of time (the machine is a fatigue tester, so it accumulates load cycles on a test sample over time).  But every once in a while, it will miss. And by miss, I mean it stalls.  The program is writing certain paramaters to a text file for review later.  What we see is a major delay in the time component of the data file.

It will look something like this.

Time:

604.150

604.200

604.250

604.300

604.524

604.554

604.575

The problem with this delay is that if the test subject is fairly stiff, the program stalls, causing the valve to stop and stay in whatever position was it's last signal (normally open).  This causes the cylinder to keep moving and normally results in unwanted damage of the test subject.  The program has checks built into that will shutdown the pump, center the valves, release the load, and unload the system pressure if the cylinder moves out of a given position or force range.  However, because the program stalls, these checks do not work until the program resumes and it is too late, the damage is done.

What can I do to eliminate this stalling problem?  Any help is greatly appreciated.

Thanks

Adam Kuiken

Senior Test Engineer

Arctic Cat Inc.

[RavensFan]

What do you consider a "major delay"?

The list you show has time stamps (I'm assuming they are seconds?) of about 50 milliseconds to start.  Then there is a jump of .224.  Is a 1/4 second considered major?  Then  30 milliseconds and 21 milliseconds.

You are running this on a Windows PC.   It wouldn't surprise me if Windows steals a 1/4 second from you every so often to do other stuff.  I see things like virus scans steal multiple seconds.  What else is going on with the PC when the program stalls?  I don't see the correlation between a stiff part causing Windows to stall.

It sounds to me that your application requires something more reliable than a Windows PC to control.  I would consider a real time operating system.  Or something like a compact Fieldpoint or compactRIO.  Or use a real hardware PID controller rather than a LabVIEW subroutine running under Windows.

[ArcticCat]

I made a typo.  The Delay is normally around a second to 1.5 secs.  That should have read 605.524. 

I had a feeling that it was something that Windows was doing in the background causing the issue.  I also figured the best way out of the box would be to upgrade to a Compact RIO and realtime control, but I was hoping that there was something I could do with the current system to get the consistancy I need.

Thanks.

[RavensFan]

Still 1.5 seconds doesn't surprise me on Windows.

[SnowMule]

Another sledder, huh?

What program architecture are you using?  Smashing data into a queue and reading it in a consumer loop can get rid of some of those delay problems.

Windows is okay most of the time but when you need strict control of timing it really sucks.  Have to move that into a dedicated system to get real tight control... either embedded, like cRIO, or FPGA.

I'd try shutting off or disabling all the non-essentials ... antivirus, windows updates, itunes, any NI stuff you're not using... go through msconfig and turn off all the fluff that windows loves to turn on. 

[camerond]

Sounds like you have to boot this computer up with minimum (or no) communication with the outside world, and run only your LabVIEW program and the other necessary bits for it. Disconnect from WiFi, Bluetooth, kill all automatic updating programs (or messages which tell you that you should update - Acrobat, Java, and of course, Windows, seem to be three of the most insistent ones), go down the list of Services (Control Panel) and disable/prevent from starting everything that is not absolutely necessary (you can find lists on the Web of services which can be stopped, and when), make sure that you don't have any scheduled disk defragmentations, tell Windows to write things to the disk when they appear, not in the background. There's a gazillion things going on in a "typical" Windows moment, you want to make sure that you have as few as possible.

If you can identify but cannot stop some competing programs, see if you can get them to warn you when they are starting, so you can set some kind of flag (or find the change of a bit on a COM port) to tell the hardware to stop immediately and stay stopped until it hears from LV again.

Cameron

[AnalogKid2DigitalMan]

Not criticizing the approach, but this sounds better suited to using dedicated PID controllers and/or a PLC for the control functions and having LabVIEW mainly acting as process setup and data collection.

-AK2DM

[camerond]

Thinking about it more, it really sounds like you need a hardware failsafe, maybe triggering when a pressure gauge hits a limit. (Even just a pressure relief valve)

Cameron

[The_Seeker]

First, you need to determine if this is a Windows problem, or if your LV code is responsible.

If your data files are large, and you are building the text file using inefficient LV methods, you may be making an unexpected call to the memory manager. This is just one possibility -- there may be others. (Restructuring the code, with the file save in an independent loop, may aleviate the issue to some extent.)

If you have eliminated the possibility of a LV-based problem, then Windows could very possibly be the culprit (as suggested by others in this thread). Windows optimization *might* get you there, but there is always some uncertainty if Windows is in the equation. In testing, it might appear to be fine, but then Windows decides to scan the DVD drive, and you experience the failure again.

Side Note: Back in the LV 2007/2008 days, I had a similar intermittent Windows problem. It appeared to be related to the DVD drive. The solution was simply to put a DVD disk into the DVD drive. This prevented Windows from scanning the drive at unexpected times, solving the problem. 

If Windows system timing turns out to be the culprit, migrating to LV RT might be the easiest path forward. cRIO is a popular choice, but I personally prefer PXI or a desktop PC configured to run RT.

If your application doesn't require the user to engage other Windows applications (Excel, Word, etc), and additional hardware costs are the primary concern, one option is to configure your existing PC to run RT. This can be a cost-effective solution if there is some reluctance to purchasing additional cRIO or PXI hardware.

Here is a link to the requirements for running LV RT on a desktop PC:

https://www.ni.com/en/support/documentation/supplemental/08/requirements-for-desktop-pcs-as-phar-lap...

We have successfully used LV RT on customized PC hardware when the application requirements exceeded the top-of-the line PXI controller, or when the cost for a capable PXI system has been prohibitive. In both of these situations, a custom LV RT PC makes good sense.

Anyway, good luck with your issue.

-- Dave