Troubleshooting hints - differences between development environment and release environment

[MikeBowen]

Hello,

I am using LabVIEW 8.6 for an application involving stepper motors.  We are using 2 Velmex stepper motor systems to run PIR targets for a manufacturing test.  The problem I am having is that the program runs fine in the states where it was developed but hangs approximately every 10th run when running in the manufacturing environment in China.  Since I am on the EAst coast there is the 12 hour time delay, and I am planning on doing a live support call to find out what is going on.  The person who installed the code made some modifications, but I am not aware of the differences.  He also may send his code here and see how it runs.

The assumption I am making (based on past experiences from here in the States) is that the stepper motor interface is hanging.  since I can't recreate the problem here I can't troubleshoot it.

With a past prooject I was working I did some code on one machine, and when I ran it on a different machine the order of execution for items in a sequence was different between the machines.  Has anyone run into this before?  Unfortunately for the task at hand the stepper motor has to run at the same time that analog data on multiple channels are being acquired.

Any suggestions?

Mike

[smercurio_fc]

You're really talking about different things here. You're also indicating a very bad (in terms of configuration management control) situation.

Let's handle the development/application environment issue. Yes, it's possible that block diagram objects that are in parallel and have no data dependency could execute in a different order between the development environment and the application environment. It's also true that this is not restricted to be solely between the development environment and the application environment. It can happen within the development environment all by itself. It's also true that, by definition, if they are in parellel and have no data dependency, then it should not (by definition) matter which one executes first. The fact that they do implies an error in your code, and not in LabVIEW. I.e., you have neglected to force a data dependency where there should be one. Note that you can have parallelism and data dependency simultaneously - they may be only mutually exclusive to a point. In your example you've indicate that the stepper motor has to run at the same time that analog data is being acquired. Well, clearly, based on your comments, at some point, operation of the stepper motor depends on some data. So, how are you establishing this temporary data dependency? Queues? Notifiers? Passenger pigeons? What kind of framework are you using? Producer-Consumer? Master-Slave?

Now, as to the configuration management control issue: Why in the world is a manufacturing environment in China allowed to make changes to source code?!? They should be running a controlled executable. Are they running the code in the LabVIEW development environment? This is bad. I also don't see how you can be expected to determine what effects the changes may have done if you don't get the changes in the first place. 

[MikeBowen]

Thanks for the reply.  I did think about putting in a producer-consumer.  It turns out that there was a problem with the computer it was being run on.  The computer was changed and the problem got better, although it did not go away.  The stepper motor was done through a com port talking to another hardware / software solution.  The tasks are truly independent and pass no data between them.  However, the other task being run involves collecting analog data and updating on a chart (and in one instance stimulating an analog output), and when the stepper motor starts the analog data collection is not updated on the chart.  I figure this could be because the charts are a memory hog.  No data is lost, and it is running, so I figure if it's not broke don't fix it.  It now occasionally locks up. I can run it here for 100 times with zero lock ups, and the last report I got was it ran 30 times in china before locking up.  I may have to start examining differences between their machine and my machine.

Now as to the configuration management.  It needs to be clarified a wee bit.  I'm doing the initial code development / updating, and then we hand off to china, which can make code tweaks as needed.  They also have 3 lines running this program, and one of the items I did was to load the differences between lines from configiration files (INI files), and that I got working well.  We have fixture variables, items that vary between lines, and configuration items, which are I/O lines and what not.  The digital I/O and analog I/O referenced MAX tasks, and that worked well.  There are also a com port call for the stepper motor, and in another program an RF power meter, which had to be referenced from an INI file.  I will reccomend that they do their code development on one machine and create executables to store on the production lines.  We have been bouncing code back and forth from the team members from the states that are currently in china doing the install.

Mike

[smercurio_fc]

Charts are not memory hogs unless you make them so. Charts have a built-in memory buffer, which is 1024 points (per plot) by default. Unless you're plotting hundreds of thousands of points, or dozens of plots, this should not be an issue. You said that the data collection is not updated on the chart, but you also said that no data is lost. How do you know that data is not lost? Are you writing it to disk? If so, is this happening in parallel to the chart updates? Have you tried removing the chart?

[RavensFan]

Do you have an electrical noise issue?  If some of the communication is relying on RS-232, but it somtimes breaks down when the motors are actually running, perhaps the electrical motors are generating noise that is interfering with the serial communication?

[MikeBowen]

My apologies for not being clear enough.  I am running the stepper motors at the same time as running the data collection (analog input and in one case an analog output also), and when the stepper motors are running  the data collection is not updating, but when the stepper motors are done the data collection completes.  This "lag" I am assuming is due to the buffers in the hardware/software.

The odd thing about this is that the code runs fine at my location but in China runs intermittent.  I'm pretty sure the chart is not the problem here, in that the stepper motors hang, and because the stepper motor hangs the program hangs.

From a debugging stand point it's difficult/impossible to debug something that can't be recreated.   At one point the program was haning every ten times, the computer was changed, and the problem got much better, though it did not go away.  The assumption I am making is that it could be a problem with the comptuer, such as an interrupt conflict.

Mike

[MikeBowen]

That is of course a possibility, but it's difficult to recreate the problem.  When our 2 gentlemen that were sent to China returned I had a conversation with them and started looking into the LabVIEW drivers.  Their interface consists of a single VI with a load of terminals that is very difficult to interface to.  One of the terminals is Timeout, and I had a value wired to it.  I also looked at their web site and one of their items was REset Driver functions, which has a disclaimer that this function is not available in LabVIEW.  Once I calmed down I actually considered writing my own drivers.

ONe other item that came up was that after the program hangs stepper motors don't recover nicely in that the motors need to be moved by hand I believe.  When we start the program we home the stepper motors until the limit switches at one end open.  What I would like to do is to check the limit switches to see if they are open.  In the command set the limit switches status can be queried, but it doesn't look like it is immediately available.  I'll have to spend some time playing around with their VI to see if I can make that determination.

Mike