slow socket server with Pacific Inst. dll

[cb3]

I have developed a LabVIEW vi that queries for channel data using Pacific Instruments (PI660DLL) library calls, called from LabVIEW library function nodes. This vi works fine and I'm able to poll the system for data every 50 msec or less. The vi is part of a system that passes data via the socket server to other vi's on other clients. If I enable a single data socket within the vi (even if it isn't passing data), or if I run another vi that has any enabled sockets, my system performance falls off to near zero. Data polling takes up to +1 sec to complete and the windows user interface response is extremely slow.

I have talked to the people at Pacific Instruments. They are at a loss as to the poor performance and claim that their code does not have any enabled networking. They have not used LabVIEW to wrap their functions.

Has anyone run into similar problems? Does anyone have an idea of where to look for this performance problem?

Thanks

[Travis_M.]

Hello,

I think the best way to approach this problem is to test whether or not the problem is related to the instrument driver or something in the LabVIEW code.  I recommend with starting by keeping your current architecture and removing all calls to the instrument driver and replacing them with simulated data.  For example, if one call to the driver polls the state of the instrument and returns some I32 value representing the state, you could use the Random VI with the multiply (for scaling) and a conversion to simulate the call.  If Datasocket still brings your computer to a standstill this problem is likely something in either your code, or the network/computer setup.  If datasocket works well, the problem is likely related to the driver or some dependency between your code/datasocket and the driver.

Hopefully this may shed some light on the problem.  Keep us posted how it goes!

Regards,

[cb3]

Travis;

Thanks for the input. I had already tried to narrow my search by substituting dummy data for the function call output. The vi works fine in this case. I did add some additional timers and have determined that if there is no socket server running, either from attached sockets on this vi or another vi running with sockets, the execution time of the Pacific Instruments dll function is approximately 25msec. If there is a socket running it is approximately 1000msec. So I'm pretty sure the problem isn't so much in my code but some interaction between the socket server and the PI drivers.

One thing that I did find that's interesting (but probably not related), is that I have the function call within a timed loop. I'm using "Wait until next ms multiple". If I set a time of 1000ms or less, the execution time of the function averages around 1000ms. If I set a time of 1500ms, I will get differing times for the function execution such that if I add a pair of times, the average will be 1500msec (ex: 1000ms/2000ms). This is not the loop time but the actual time to execute the function. Screen performance also degrades during this slow down.

There may be a system configuration problem, but I'm not sure where to look. This computer has 4 CPU's so it should be plenty fast enough.

Thanks;

cb3

[Travis_M.]

Hello,

This is an interesting problem.  I suspect that with 4 cpus there might be a problem with some sort of non-threadsafe DLL somewhere along the way that is being scheduled causing some sort of hang-up.  If you change the timeout value of the timed loop, is the average execution time always roughly equal to the timeout value?  What happens if you create a simple VI designed just do basic datasocket?  I assume that this runs fine until some of the instrument drivers are loaded, but what about taking this basic datasocket VI and gradually including more and more of the instrument driver in an attempt to determine if the behavior is limited to a single component of the driver.  We could also disable multithreading in LabVIEW to see how that affects the performance (under Tools->Options->Performance and Disk).  I assume that you are using the most recent drivers but it never hurts to double check that too.....

Hope this helps, let me know how things go!

[cb3]

Travis;

I've paired down my vi considerably so it's easy to segregate the problem area and have pretty much narrowed down the library functions that I am calling that cause the poor performance. It's not just any function call in the Pacific Instruments (PI660DLL) library, but the calls that actually report back the real time channel data.

I have been able to narrow down the things that do cause the performance to go bad.

• With socket servers disabled the calls to the PACIFICDLL functions respond very quickly. A single query of 11 channels to PI660DLLGetManyEUValues takes 25 to 40 ms.
• If I enable the socket server the same calls to PI660DLLGetManyEUValues take about 1000 ms. There is no apparent effect on cpu loading but the mouse/screen interaction also becomes poor.
• The performance degradation occurs with calls to both PI660DLLGetManyEUValues  and PI660DLLGetEUValue. A single call to PI660DLLGetEUValue takes as long as a single call to PI660DLLGetManyEUValues for 11 channels. If I call multiple times for 11 channels it takes approximately 11 seconds.
• Calls to PACIFICDLL functions that do not return Eu's, such as PI660DLLGetUnits, are not effected by the socket server.

I've tried disabling multithreading, but this does not change the performance.

There are some interesting things going on as far as timing issues go. First, no problems at all if sockets are disabled. I get what I'd expect. Next, if I start out with a "Wait until next ms multiple" of 2000 ms or greater for my queries, the execution time of the function is approx. 300 to 450 ms. If I decrease the wait time incrementally to 1000 ms, execution time remains at 300 to 450 ms. Average loop time remains the same as the specified wait time. Once I drop below 1000 ms, even by 1 ms, both the loop time and the function execution time bounce from 990 ms to 1016 ms (avg 1000 ms). Simply raising the wait time back to 1000 ms does not rectify the timing. I have to raise it to 2000 ms before I see the times even out to what they were when I started. The CPU is hardly working during all of this.

One of my biggest problems is that there are 4 vendors (including NI) involved with this system. Each of which has written software.

If you can think of anything else, it would be appreciated.

Thanks;

cb3

[Travis_M.]

Happy Friday!

One question I have for you is about how you are using Datasocket.  Are you using the Datasocket VIs or are you sharing a data item via right clicking a control and choosing to connect to a Datasocket item?  In either case, you need to have the Datasocket server running to communicate – does the slowdown happen when you turn on the server, or when you have Datasocket VIs in your program?  I know it seems unusual that the VI can behave in one way (say with a 1001 ms wait), then behave badly (say with a 900ms wait), then continue to behave badly where it used to work (say by going back to 1001 ms wait).  This can happen for any number of known and unknown reasons, but one possibility is that the hang-up could come from the driver.  Anytime (well most of the time…) a DLL is used in LabVIEW, the DLL is loaded into memory.  The state of the DLL is preserved between calls and is not reset until the DLL is unloaded when the program ends.  This could mean that some condition in your code is putting the driver DLL in a bad state that it is not recovering from until you set the wait time back to 2000 ms.  Additionally, the fact that your CPU is not busy further indicates that this is likely some sort of resource allocation/conflict problem (though I have NO idea why it would lock up your UI). One possibility for troubleshooting is to remove Datasocket from the picture completely and use the underlying TCP/IP VIs to send data (there are also UDP functionality which you can use).  It’s no trade secret that Datasocket uses TCP/IP to transmit/broadcast data – removing the extra NI software associated with Datasocket may shed some additional light on the situation (though you may end up having to spend more time than you are comfortable with converting your program to use TCP.  One tip is to start with a small VI which can reproduce the problem and try it with that before moving on to rewriting your program….

I don’t really have any additional ideas at this point, except to continue the debugging process and eliminate as much as possible.  Please keep us up to date with any new information available.

Have a great weekend!