Remote panel access stops after 1 second - client could not keep up with server

[Volker64]

I have build a rather big project in labview 2018. It is running well on my Cdaq controller under linux RT. I managed to create the executable which includes the html-file that is needed for remote access. When I connect to the remote panel using the internet explorer by http://xxx.xxx.xxx.xxx:8000/abc.html the connection is established for approximately 1 second. I can see the front panel of the application and if I am quick enough, I can even press a control button of the front panel. Then the panel disappears and I get the message that the "Server has disconnected the client because the client could not keep up with the server".

The same problem occurs when I try to connect to the remote panel directly from labview. I use a german labview development system. Please see the german error message attached.

I found the following https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z000001DdGSSA0&l=de-DE . So far this did not help me. My project is only using an event structure and timed loops - no while loops without wait. I found the ni-rt.ini file on the RT target in the path /etc/natinst/share and I added the token RemotePanel.MaxMsgBacklog = 1000 and even 10000, but it did not result in any difference.

Has anyone a good idea?

[cbutcher]

Can you share the VI you're connecting to?

[Volker64]

I attached it. There are many sub-vis that I don't want to upload.

I played around today. If I create the html-file for my laptop then I can see the frontpanel in my laptop's browser while it is running in the labview development system. No Problem.

If I create the html-file for the cdaq and include it in the build then I can only see it for one or a few seconds before I get the message that I mentioned before. Funny enough: If the application is running under high load doing data aquisition, then the remote panel stays visible up to 5 seconds. Without load it shows up for less than 1 second.

I have tried to put "waits" for 100 mm in every timed loop, but this did not change anything.

[RavensFan]

A few problems I see is that your block diagram is absolutely HUGE.  There seems to be an abuse of local variables.

Even the front panel seems oversized with an ugly font.

If I had to guess, your VI is overloading the resources of your cDAQ and that is why it can't show the front panel.

[Bob_Schor]

Can you describe the layout of the LabVIEW Project?  I'm only now getting into doing LabVIEW RT on a cDAQ, but I do have experience with a PXI-box and PXI 88xx controllers.  In those routines, all of the UI code resided on the Host (PC), while all of the RT stuff resided (naturally) on the PXI, with the PXI controller being the Remote Target.

I'd started developing this code several versions of LabVIEW earlier (my earliest version, I think, ran on LabVIEW 2012), when the Conventional Wisdom (as I knew it) was that the Top Level RT code had no Front Panel Controls.  I used a QMH Model for driving the RT side, and used a Network Stream to allow the Host to send RT Messages to the RT QMH when it needed to tell the RT side to "do something" (there was another Network Stream that the RT side used to send similar "Please do this for me" requests to the Host).  Needless to say, the very first thing the RT program did was to initialize its part of the Network Streams (the Host also had an initialiization routine for its end of the Streams), after which the Remote detached (using Start Asynchronous Call) a loop that waited for messages from the Host to the Remote and fed them to the Remote QMH.

This worked surprisingly well!  Ordinarily, we would start the PXI code running (and leave it running for days), happily waiting for the Host to connect, then fire up the Host when we needed to collect data, the Host would reach out, the Remote would establish the connections, both would send up their respective Queues to pass the QMH Messages from "the other side, intended for me" to the QMH, and (forgive the phrase) "Bob's your Uncle".  When the Host said "That's All, Folks!", it sent a final Message to the Remote, which closed all of the Network Streams, then basically rebooted the PXI, causing it to auto-restart the Remote's routine.

Bob Schor

[cbutcher]

Hi again,

Thanks for uploading the VI.

I initially thought RavensFan could probably have been a bit more helpful with some points, but having opened your VI I agree completely - my computer lagged just trying to move around the block diagram using the navigation window when it wasn't running.

I can see you have at least two timed loops running at 1kHz, which whilst not inherently problematic for process variables, is likely a problem for updates - since you have many controls (via local variable no less) in these loops, they are being updated very frequently, which will create a large number of update messages to be sent to your remote panel. 

The best thing to do is work to reduce the size of this VI - start in one place and just start cutting away with subVIs and clusters etc.

As a temporary fix, you might look at reducing the rate of the "DataDisplay" loop, but I have no idea if that will help (and perhaps you described already trying this without results) because I can't see what's going on in that loop very well.

[Volker64]

Hi everybody and thanks for the replies.

@RavenFan: You are correct with regards to the size of the blockdiagram and the frontpanel and the abuse of local variables, but this is owed to my limited knowledge when I started this project. If I'd start today, I would approach in a different way. If a nicer font can help, then please let me know which is the best to choos and I will give it a try. Please see attached a screenshot of the frontpanel when the application is running (with simulated data). I use the captions instead of the control names, because this application is loading the captions from a CSV-file while running to make it multilingual. I like the font.

@cbutcher: Believe it or not. the project is running very nicely on my laptop and on 2 cdaqs with Linux RT, where one of them is a 2core machine only. The loops are running at fixed speeds of e.g. 1 or 4 Hz. One other loop is running at a frequency which is calculated within the project and depends on the amount of data that is generated. This again depends on the timing of the daqmx-timer which can be adjusted, but the frequencey is in a range below 10 Hz.

With a 2core cdaq (9136?) I manage to run the application at 250 Hz daqmx-timing with 2 NI9361-modules using a total of 16 counter channels. The load per core stays underneath 50%. The system is running at one of our customers for 4 weeks non-stop at 100 Hz daqmx-timing without any issue.

Anyway it does not make a difference with regards to my webserver-problem when I run the daqmx-timing and all the other loops at lower speeds. I tried that already.

As mentioned above I fully agree that I should tidy up my project and etc., and I also come to the conclusion that it is probaply an overload issue of the Webserver, but I think that the performance of the webserver does not cope with the performance of the rest of labview.

The reason for my Webserver-experiments is that I want to be able to check whether my application is running well at our customers after software modifications. These customers reside all over the world. As my Webserver-Approach does obviously not succed, I will give x11vnc a try. I just found this yesterday:

https://forums.ni.com/t5/NI-Linux-Real-Time-Discussions/Installing-and-using-x11vnc-on-a-2016-x64-ta...

Or is there another VNC-server that is better/easier? I will come back with an update once this has been tried.

[cbutcher]

Sorry, to reiterate - I wasn't suggesting that your hardware would be unable to run the loops at those rates (because clearly it can) - only that if you update indicators at that rate, it will generate a lot of separate data transfers to the remote panel, overloading it.

If you don't connect a remote panel, presumably the indicators are not updated (because there isn't a need, if nobody is watching?) or similar, and so no problems. This may not be technically accurate, but I think the behaviour described is how it goes. Maybe they're updated but there's some special headless way to avoid a UI update cycle, or defer panel updates is set true always if nobody watches? Speculation only here...

I suspect this is related to why if I remotely connect to the cRIO I use after it started running, any indicators that are only updated on startup are blank. I need to write them after I connect to see a non-default (e.g. non-empty or non-zero in many cases) value. Putting them in a loop solves this (inefficiently, but oh well), but I have far fewer things to check than you have.

[Volker64]

Hi. I have installed X11VNC and made it autostart. On the windows pc I use TightVNC to connect.

This works very nicely and is very performant - in a local lan as well as via internet by port forwarding.

For me this is a perfect solution.