Best practices for file I/O within producer/consumer loops

[jamoore84]

I'm looking to add file recording and playback functionality to a pre-existing data collection program.  The original progam is based on a Moore-style state machine, which I have added four additional states to.  They are: Record Start, Record Stop, Playback Start, and Playback Stop.

What I have done, and what has since been identified as poor programming practice, was to "initialize" (either create or load) the appropriate file within the state machine loop during the "Start" command (for record or playback functionality), and then provide the file reference as an indicator, which is linked to for the appropriate read or write operation(whether I'm playing back or recording).  The actual I/O occurs within the the Consumer Loop. (screenshots attached).

This is my first labview project outside of tutorials or other small examples, so any advise and constructive criticisims are welcomed.  Specifically with regards to file IO and refnum routing (it gets a little hairy in the consumer loop)!

I'm running Labview 8.6 on Vista business.

[Ben]

Any time you have multiple threads that can write to the same object (control Indicator, local global...) AND the writing is not protected by mutexes etc. you have a potential race condition.

Please review this Nugget on Action Engines to see how an Action Engine offers an easy method to share data with the work of acquiring mutexes etc.

Ben

Message Edited by Ben on 03-12-2009 03:00 PM

[jamoore84]

Ben,

Thanks for the suggestion.  I think it's a little outside my ken at this moment, but  I'll look it over.  Despite any grevious coding transgressions, I have experienced some limited success with the current setup.  While the use of Action Engines/Functional Globals may constitute the best practice, I might revise my post to read "acceptable and/or easily absorbable practices" instead.

Are there any other opinions on this?  Let me start by listing a problem and posing a question:

Problem:  I am able to playback a file only once.  Subsequent attempts at file playback do not work.

 Question:  Is the current implementation of file reference routing -- initially using the "Read in" reference to feed the Read From Binary File function and then looping the "Read Out" file reference for all subsequent read operations -- acceptable?  Also, the use of my FRC/FPC local variables (which stand for  First Record Click and First Playback Click, respectively) seem a little ad hoc.  I'm using them to control which file reference gets routed to the file read/write function.  The first time the call to read/write is processed, I want the reference from the file initialization state to be routed (from the State Machine).  all other read/write iterations will use the file reference that comes out of the read/write binary file function.

I started off using the First Call function, but this would not allow multiple file playbacks over the runtime of a VI.

Are there other opinions on how to do this?

Thanks in advance,

jimmy 

[BCho]

Hey Jamoore,

Ben makes a good point warning about race conditions and points to a very informative resource. Though it is a little difficult to understand the entirety of your program from the images you provide. The main reason for this being we can not see every case of the state machine or the entire producer consumer loops. However, it appears that you are trying to use local variables to control your producer consumer loops. A typical producer/consumer architecture uses queues to pass data between the two loops. Take a look at Applications Design Patterns: Producer/Consumer for a little more information on the what I am referring two. I can not be positive that this will solve all of your problems, but it is certainly something to consider. For instance, instead of passing your file references through local variables, you could do the same with the queue.

Hope this helps.

-Ben

Applications Engineering

National Instruments

[Ben]

---

jamoore84 wrote:

Ben,

Thanks for the suggestion.  I think it's a little outside my ken at this moment, but  I'll look it over.  Despite any grevious coding transgressions, I have experienced some limited success with the current setup.  While the use of Action Engines/Functional Globals may constitute the best practice, I might revise my post to read "acceptable and/or easily absorbable practices" instead.

Are there any other opinions on this?  Let me start by listing a problem and posing a question:

Problem:  I am able to playback a file only once.  Subsequent attempts at file playback do not work.

...

Thanks in advance,

jimmy 

---

HI Jimmy,

I don't give up easy.

Let me try to exaplain the issue with race conditons with a contrived example, the

 "Command by Mail box" case.

Imagine you had a job where you recieved your orders via an old fashioned mail box. You never really saw your supervisor but relied on getting orders via the mail box. Now imagine the mail box could only hold one message and any time a new message was inserted, the old one would fall into the trash.

So you come to work each day check your mail box do what was ordered and everyone is happy.

The next day you come in and without your knowing, you are assigned to do the work of two bosses. So as long as you check your mail box more often than the two of them assigne work everything is fine... until you take a day of vacation! So you come back in the day after vacation and check you mail box and work a way until you catch hell for ignoring orders !?! Wel it turn out the order from boss 1 was replaced by the order from boss 2 while you were on vacation. Oh bother!

How can we fix it?

1) Expand the mail box so it hold more than one order. You just process them in the oder they are recieved.

2) Change to mail box to not accept a new order until you have removed the old.

Now back to reality!

Local variables act like the funky mail box. The last message insterted over-rides the previous.

Multiple variable writer ar like multiple bosses.

Queues operate like an expanded mail box, letting you handle each message in order.

Action Engines operate like the "mutexed" mail box.

Why I don't want to encorage you to "just patch up" what you have.

All of the less than ideal solutions either over-sample (Check e-mail twice as often as bosses assign work, waste CPU, an exercise in futility if you are coding in a non-Real-Time envirionment like Windows) or use a mutex to control access to the shared resource ( in this case local varialbes).

LV offers mutexes through semaphores (found on the syncronization pallete) but...

WHY WORK SO HARD?

In my AE Nugget I explain that the exection of an AE is automatically protected by LV. So in the long run it will actually be easier to learn how to use the AE programming construct than it will be to learn how to solve the problem without them.

so GO FOR IT! Take the lazy route and learn how to use the AE construct. Use the Syncronization pallete for Queues.  

Just trying to help,

Ben

[jamoore84]

Ben,

Thanks for your second post.  You and bcho both mention using queues, which I do use for the raw data transport. I also remember previous post of yours (the link of which escapes me) which recommends the use of a queue over an AE if both are appropriate.  I've never considered routing a file reference into a queue and am not sure how that would work.    Would I replace the file reference indicators by enquing/dequeing  them from a  "Playback/Record Queue"?

As far as AE's are concerned, it seems a subvi with case structure states "Playback Start", "Record Start", "Record", "Playback", "Close Record File", and "Close Playback File" states might be the right path to start down.  I'll try to work out some examples and see where it takes me.

Are there any more tutorials or examples on AE's I can browse?

Thanks for all your help,

jimmy

[Ben]

---

jamoore84 wrote:

Ben,

Thanks for your second post.  You and bcho both mention using queues, which I do use for the raw data transport. I also remember previous post of yours (the link of which escapes me) which recommends the use of a queue over an AE if both are appropriate.  I've never considered routing a file reference into a queue and am not sure how that would work.    Would I replace the file reference indicators by enquing/dequeing  them from a  "Playback/Record Queue"?

As far as AE's are concerned, it seems a subvi with case structure states "Playback Start", "Record Start", "Record", "Playback", "Close Record File", and "Close Playback File" states might be the right path to start down.  I'll try to work out some examples and see where it takes me.

Are there any more tutorials or examples on AE's I can browse?

Thanks for all your help,

jimmy

---

Re: "both appropriate"

I still feel that way but I am not sure both would end up doing the same thing. WE would have to work out the messaging then compare with the AE. If after that they are the same, I save myself work by using the queues. But it looks like you have already started to think AE

AE approach

Those anmes sound good. What you want to do when implementing those action is make usre all of the logic associated with those action are handled inside the AE and any state info (like file ref, paused/recording) get saved in the Shift register before teh AE returns to the caller. Think of your AE like it is a cassette recorder. You are only allowed to push the button on the front panel. If you need more functionality add a new button.

When I design these I start out with

1)the names as you have done.

2)Desribe what each action does

3) Make sure all info required to perform the actions are inside the AE. In your case a file reference is one of these values. Another "state" would be be "record vs idle vs playback".

4) Review 1-3 to stream line and make sure they are not contradictory.

5) If the AE is really complicatd I work out the typical usage and track the sate ON PAPER to make sure.

6) Develop AE along with stand-alone tester.

I hope some of that helped.

Ben

[jamoore84]

Ben,

Thanks for the pointers.  I've attached an Action Engine inspired by your Moving Average example , but I'm not sure what to do with all the unwired terminals.  For cases where I am closing the file reference, I don't think it  makes sense to string it across.  From reading forum posts on whether or not to enable the "Use default value if unwired" setting, the majority opinion seems to be not to do this.  Does this extend to file references, too?

---

3)...Another "state" would be be "record vs idle vs playback".

---

Could you elaborate on this state?  I don't immediately see what this would be used for.

Thanks again,

Jimmy

[Ben]

---

jamoore84 wrote:

Ben,

1)

Thanks for the pointers. 

2)

 I've attached an Action Engine inspired by your Moving Average example , but

3)

I'm not sure what to do with all the unwired terminals.  For cases where I am closing the file reference, I don't think it  makes sense to string it across.  From reading forum posts on whether or not to enable the "Use default value if unwired" setting, the majority opinion seems to be not to do this.  Does this extend to file references, too?

4)

---

3)...Another "state" would be be "record vs idle vs playback".

---

Could you elaborate on this state?  I don't immediately see what this would be used for.

Thanks again,

Jimmy

---

1)

No problem!

2)

No enum posted. No problem.

3)

Unwired ....

For tunnels that maintain state info (like your file ref) wire those across or use the "Right-click >>> Linked Input Tunnel" option. This will enusre that if your file has been opened the order in whcich the action are invoked will not result in errors. EG If you have selected "use default if un-wired" then if you add new action latter and forget to wire the tunnels you may loose your file ref after the new action is executed. By not using "use default if unwired" you will be reminded to wire the tunnels (unles you have linked tunnels).

For outputs tunnels (not used by Shift Registers) that return a value to the caller via a indicator, then using "Use Default if Unwired" is fine in my book.

4)

I was just throwing out ideas so don't worry if your model does not fall out the way I outlined.

When we started this thread you gave us only breif glimpses of your code so I can't say what you have posted will solve your challenge.

Could you please post a simple "Tester" (along with the enum) that demonstrates how you intend to use this AE?

This would give us some context to help advise you further while also qualifying this thread as an appendix to my AE Nugget.

Still trying to help,

Ben

[jamoore84]

 

Could you please post a simple "Tester" (along with the enum) that demonstrates how you intend to use this AE?

This would give us some context to help advise you further while also qualifying this thread as an appendix to my AE Nugget.

---

Ben,

   Attached are the Action Engine,it's enum, and the AE tester.  One more enum to follow with the next post.

These files are a mockup of the file IO functionality I want to drop into my larger program.  The tester is a general outline of my larger program.  If I can get these working, then I'll be good to go.

As it is right now, though, I have hit a snag with Recording that I cannot immediately diagnose.  Rather than spin my wheels I was hoping someone more knowledgeable could take a look and hopefully see the issue right off the bat.

Comments and criticisms welcome.

jimmy