Minimize runtime engine overhead by keeping the runtime engine running.

[RHryniowski]

Hi community, 

I have a small .xml parser that needs to run many, MANY times whenever a new file is generated and I'm looking for ways to reduce the LabVIEW Runtime engine overhead.

Are there techniques to keep the runtime engine running in the background so it doesn't have to load every time?  Can I launch an actor that "holds the door open" (++if I can call it Hodor.vi) to keep the engine from unloading without stopping the main executable from terminating?

[altenbach]

Why not design the parser so it keeps running?

---

@RHryniowski wrote:

to keep the engine from unloading without stopping the main executable from terminating?

---

What is the "main executable" in this context? What is the meaning of "without stopping the main executable from terminating"? very confusing!

[billko]

Can you just have the parser do everything at once to the new file?  Why does this parser need to be opened multiple times on the new file, instead of taking care of everything at once?

[JÞB]

Just set your application to run as a Daemon .  Leave it running. 

Of Course, that just begs us to question why you would be generating an xml file in the first place!  Parsing xml requires a LOT of overhead calling into the w3c published DOMUserDefReff.dll that probably takes more processor allocation and collection than the lvrte.

[RHryniowski]

---

@billko wrote:

Can you just have the parser do everything at once to the new file?  Why does this parser need to be opened multiple times on the new file, instead of taking care of everything at once?

---

Sorry, not being clear.  As the line runs new .xml files are being constantly generated at a rate of 1 ever 3 seconds or so.  The files are being generated by a blackbox executable called by the equipment.  I need to consume and move files as they are generated and the ~2 second overhead the runtime engine needs to start up is killing me.  I've through about an old ActiveX server that would run in the background and just be triggered by the equipment creating the .xml file, but that seems too inefficient and I worry about what happens if it crashes or locks up.

To keep the runtime loaded, I've thought about launching an actor, or a second executable (checked for active when the parser is called, won't be run a second time if it is already active) that just runs an infinite loop independent of the executable processing the .xml files, has anyone come up with a better way to keep the engine from unloading?

[billko]

So are these instances launched manually?  If not, you already have a way to detect when new files are generated; could you move the detection code into the parser and just have it detect a new file and parse?

[RHryniowski]

---

@JÞB wrote:

Just set your application to run as a Daemon .  Leave it running. 

Of Course, that just begs us to question why you would be generating an xml file in the first place!  Parsing xml requires a LOT of overhead calling into the w3c published DOMUserDefReff.dll that probably takes more processor allocation and collection than the lvrte.

---

The .xml are part of the original design, I had to process 525,000 old records when I started this project, took my poor laptop all weekend to do it.

So if I go the Daemon route I would leave a simple top level running that would check every second for a new file and if it found one it would launch the parser?  I'd like to keep the launcher separate from the parser for reliability reasons, over the 525,000 records above I had malformed .xml files 4 times that created problems for the Python I use to parse the .xml.  So should I structure the parser as an Actor that can be launched for each new .xml that is found and won't wait for the a parallel parser to finish?

[FireFist-Redhawk]

---

@RHryniowski wrote: So if I go the Daemon route I would leave a simple top level running that would check every second for a new file and if it found one it would launch the parser?

---

If on Windows OS, you can detect file additions by using .NET events instead of by polling, and your parser can be the callback VI that occurs when a FileWatcher event fires. Pretty slick example of an event-based file monitor here.

[JÞB]

---

@RHryniowski wrote:

---

@JÞB wrote:

Just set your application to run as a Daemon .  Leave it running. 

Of Course, that just begs us to question why you would be generating an xml file in the first place!  Parsing xml requires a LOT of overhead calling into the w3c published DOMUserDefReff.dll that probably takes more processor allocation and collection than the lvrte.

---

The .xml are part of the original design, I had to process 525,000 old records when I started this project, took my poor laptop all weekend to do it.

So if I go the Daemon route I would leave a simple top level running that would check every second for a new file and if it found one it would launch the parser?  I'd like to keep the launcher separate from the parser for reliability reasons, over the 525,000 records above I had malformed .xml files 4 times that created problems for the Python I use to parse the .xml.  So should I structure the parser as an Actor that can be launched for each new .xml that is found and won't wait for the a parallel parser to finish?

---

The underlying dll call to DOMUserDefRef that xml uses because it follows the Document Object Model is single threaded (blocking.) So is file access! (yeah, you can only write to the disc through one data bus at a time.) So, trying any type of parallel operation is not going to work.  You will have to appropriately throw and handle any errors that a malformed file causes. 

[DerrickB]

Each executable is basically it's own run-time engine. No executable, no run-time engine to leave open. If you need reliability for headless operation I can think of a two executable setup. The first that monitors for new files, perhaps using the slick windows filewatcher examples that were linked above, and communicates to a second app that always runs over TCP or some other mechanism to tell it what file to process. Then if the second app crashed the first app can note that there's a problem with the file and relaunch the 2nd app and skip that file in the future.

Ideally there could be better error-handling implemented and this all could be done in a single executable that just doesn't crash because of a parser error.