deterministic timing getting out of sync

[Mike_505]

Thanks Bill, As you can probably tell, I'm learning through the school of hard knocks....Coming from  ATE programming environments, I do wish there were boxes that I could just write some code in. It many cases it would be easier to read, eliminate spaghetti (in it's own box or not), and easier to edit.  Anyway, so far as my general methodology has been, good or bad, is to make a "little box" if I believe I'll be iterating that functionality throughout the design. Or, and much more rarely,  if it's really needed to make the design more readable.

[billko]

---

@Mike_505 wrote:

Thanks Bill, As you can probably tell, I'm learning through the school of hard knocks....Coming from  ATE programming environments, I do wish there were boxes that I could just write some code in. It many cases it would be easier to read, eliminate spaghetti (in it's own box or not), and easier to edit.  Anyway, so far as my general methodology has been, good or bad, is to make a "little box" if I believe I'll be iterating that functionality throughout the design. Or, and much more rarely,  if it's really needed to make the design more readable.

---

LOL that's one BIG reason to make a subVI that I forgot to mention... 😄

[Bob_Schor]

Is your code running on a PC or on an RT Target (running an RT OS)?  On an RT Target, Timed Loops are treated "special" to ensure that they run as deterministically as possible.  I'm not sure that this is necessarily true on a PC, especially since Windows has its own "rules" that take priority.

The Timed Loop has (I don't know what to call them ---) those "terminal things" in the upper inside corners.  If you investigate, you'll see that you can detect if a loop missed a tick.  I think that on an RT system, the Timed Loop gets "locked" to the clock -- you may miss a tick, but I think the time intervals will be multiples of the tick times (i.e. if you are running a 1 Hz, you can be a second off, but not half a second).  [I'm prepared to be told I'm wrong about this ...].

In the Tutorials and discussion of Real Time, a lot is made of keeping the code in the Timed Loop "lean and mean", and only putting in there stuff that must be done every delta-t (like taking a sample).  Once the time-critical stuff is done, the data are shipped out (usually via a Queue, sometimes, for really the fastest response, by a Real-Time FIFO) for other loops to handle.  This would certainly include any I/O -- you can use the Queue as a "buffer" to accomodate the variation in speed that comes with writing to disk.

Bob Schor

[Mike_505]

Is your code running on a PC or on an RT Target (running an RT OS)?  On an RT Target, Timed Loops are treated "special" to ensure that they run as deterministically as possible.  I'm not sure that this is necessarily true on a PC, especially since Windows has its own "rules" that take priority.

The Timed Loop has (I don't know what to call them ---) those "terminal things" in the upper inside corners.  If you investigate, you'll see that you can detect if a loop missed a tick.  I think that on an RT system, the Timed Loop gets "locked" to the clock -- you may miss a tick, but I think the time intervals will be multiples of the tick times (i.e. if you are running a 1 Hz, you can be a second off, but not half a second).  [I'm prepared to be told I'm wrong about this ...].

In the Tutorials and discussion of Real Time, a lot is made of keeping the code in the Timed Loop "lean and mean", and only putting in there stuff that must be done every delta-t (like taking a sample).  Once the time-critical stuff is done, the data are shipped out (usually via a Queue, sometimes, for really the fastest response, by a Real-Time FIFO) for other loops to handle.  This would certainly include any I/O -- you can use the Queue as a "buffer" to accomodate the variation in speed that comes with writing to disk.

Hi Bob,

Yes, I am running it in Windows. I'm not even sure what RT OS is? Why is Timed Loop a feature in my Windows based version of LV if it doesn't work correctly? When I first used a Timed Loop I didn't know it was any sort of special feature. It was just an option in Structures or when you right click on a while loop.

I have used Timed Loops before with reasonable success. However, I never tried it in a program that needs to run for days and/or weeks. You are correct that the main item I found to keeping it working correctly is that whatever is being done during the iteration MUST have a huge window within the Timed Loop to insure it has completed. As you're probably aware, LV isn't exactly the fastest... I also talked to a friend at a different company who informed me they haven't had the best results with Timed Loops either. They went as far as having someone from LV come out, which turned out not to be much help. 

In my current case I've tried increasing the time all the way to 30 seconds and still have the problem. Also, in Configure The Timing Loop I unchecked Maintain Original Phase and selected the <reset structure at start> and these seemed to have improved it's stabilization. My program will run for a couple of days before getting the error. Prior to this I would have problems within a few hours. 

As my sample data shows, when there's a glitch reading there's always an associated time glitch. The read/write VI's have been in use for years w/o every having a problem like this. I don't believe that this is the issue. What I think is happening is when the timing goes off LV discards any data that's was collected during that iteration. 

Which just gave me a thought. Perhaps I need to completely get out of the loop for the <reset at structure start> to work properly. I could add a while loop around this. It'll throw the timing of a little, but may still be good enough for government work

[Sam_Sharp]

As far as I understood it, the reason timed loops are allowed on Windows is so that you can write code that will work on RT Targets (e.g. PC running LabVIEW Real Time OS, CompactRIO, PXI) and also run on Windows. Windows is not a deterministic OS and as such, it cannot guarantee any sort of determinism on timing - it only has timing resolution to 1ms and the jitter is unbounded (if your virus scanner kicks in, it could take a second or two before anything else happens). A real time OS is a special OS that offers determinism with a fixed/known amount of jitter and the timed loop on LabVIEW RT allows LabVIEW to take advantage of these features of the Real Time OS (priority, determinism, fixed jitter).

In fact, timed loops on Windows actually require more resources and offer worse performance than a regular while loop.

To get accurate timing, you need to use a Real Time OS. I don't think that fiddling around with the settings of the timed loop will make much of an impact.

Particularly as you are writing to files within your loops - this could cause an issue (e.g. if you have mechanical drives it might take a few seconds for the drive platters to spin up if it's gone into low-power mode). Put the read data into a queue and then dequeue it and log it to the file in a separate loop. Take a look at the producer/consumer architecture example for an idea of how to do this.

[Bob_Schor]

OK, if you are running on Windows, simply use a Consumer/Producer pattern, with the Producer being a While loop with a Wait until Next ms Multiple (not Wait (ms)) governing the loop (this way, if one loop takes, say, 130% of a tick to finish, and the next one is quick and get done in 10% of a tick, you'll be synchronized again with the third tick).  One thing you need to do with this function, however, is to let it run once before you start using it "for real", as the first time interval may be short.

As you generate values in your Producer (a "timed" While loop), put them on a Queue and "export" them to the Producer loop.  It should also be a While loop with a Dequeue element (and a reasonable timeout so it doesn't "hang" when the program finishes) -- wire the True from the timeout to whatever code you need to process the data, and the False to "do nothing".  Note that, on the average, the Producer needs to be faster than the Consumer, but because the Queue buffers the data, any single loop of the Consumer can "consume" multiple Producer loops with no loss of data (they're sitting there safely in the Queue.

Try that.  I can tell you that we have successfully handled sporatic video feed (we take 10 second videos every time an "interesting event" happens) from 24 cameras, simultaneously, with no loss of data.  On a Windows PC, yet.  In LabVIEW.  Piece of cake.

For what it is worth, a Real Time OS is what runs inside some NI devices, like Compact DAQ, RIO, PXI, etc.  There is no Windows.  There is no Anti-Virus.  The OS is there to serve you and get your data as deterministically as it can (you control who interrupts whom).

Bob Schor

[Mike_505]

OK, if you are running on Windows, simply use a Consumer/Producer pattern, with the Producer being a While loop with a Wait until Next ms Multiple (not Wait (ms)) governing the loop (this way, if one loop takes, say, 130% of a tick to finish, and the next one is quick and get done in 10% of a tick, you'll be synchronized again with the third tick).  One thing you need to do with this function, however, is to let it run once before you start using it "for real", as the first time interval may be short.

As you generate values in your Producer (a "timed" While loop), put them on a Queue and "export" them to the Producer loop.  It should also be a While loop with a Dequeue element (and a reasonable timeout so it doesn't "hang" when the program finishes) -- wire the True from the timeout to whatever code you need to process the data, and the False to "do nothing".  Note that, on the average, the Producer needs to be faster than the Consumer, but because the Queue buffers the data, any single loop of the Consumer can "consume" multiple Producer loops with no loss of data (they're sitting there safely in the Queue.

Try that.  I can tell you that we have successfully handled sporatic video feed (we take 10 second videos every time an "interesting event" happens) from 24 cameras, simultaneously, with no loss of data.  On a Windows PC, yet.  In LabVIEW.  Piece of cake.

For what it is worth, a Real Time OS is what runs inside some NI devices, like Compact DAQ, RIO, PXI, etc.  There is no Windows.  There is no Anti-Virus.  The OS is there to serve you and get your data as deterministically as it can (you control who interrupts whom).

Bob Schor

Hi Bob, Thanks for this tip and explanation. I believe I tried using the Wait Until Next ms Multiple before, but perhaps at the time didn't fully understand how it worked and was not happy with the results. I then moved on and forgot about it. I setup a quick example and so far so good. Even better is that I can easily transform my program from the timed loop to this method.

Hopefully this is the answer I've been needing.

[Bob_Schor]

As I tell BME students who want to use LabVIEW for a project but don't have much experience with it, "Find yourself a LabVIEW Guru and apprentice yourself for a week".  I usually then volunteer, but so far, when they show up at my office, they only want to spend 10 minutes and only want "answers" ...  Sigh.

BS

[Mike_505]

LOL! I would love to have that opportunity. I'm the person solely responsible for all the programming in the building, ATE, NI, and even some Java. Yet, hell will freeze over before I even get to a training course.... I

[Sam_Sharp]

Well it's worth bearing in mind that there are lots of free training materials and I believe if you have an active SSP you can get access to the self-paced online training which covers Core 1, 2 and 3 and many of the other NI training courses.