thread safe queue problem with realtime

[kander1]

Hi,

I'm using Labwindows/CVI realtime and having a problem with thread safe queues.  I have sample code that shows the problem.  Basically the code initializes the queue and launches a thread to write to the queue and a thread to read from the queue.  If everything is working OK, you will see the letter "x" printed out continuously (which indicates it is writing and reading properly).  The first time I run the code it seems to work fine.  When I shut the system down (using "simulate RT shutting down") and start it up again I get an error message that the queue is full.  It doesn't matter if I flush the queue or not.  When I reboot the RT machine it works fine again until I stop and restart the program. 

I can't post the example code here because it exceeds the 1000 char limit.  If you email me I can send it to you.

Thanks,

Kevin Anderson

Lockheed Martin

kevin.anderson@lmco.com

[Mert_A.]

Kevin,

On RT systems, the OS is the only process running, and any threads in your application are part of this process.  When you simulate RT shutting down, it gives your application's main thread a heads up that it is time to clean up and stop executing.  Other threads that have been spawned as part of your application, however, do not automatically get stopped; they continue to run in the OS process, unless your main thread cleans them up.

It sounds like you are not cleaning up your threads, so that when you start your app back up, the queue is already full.  You should call CmtExitThreadPoolThread for your reader/writer threads before exitting your main thread.

Let me know if this does not solve your problem.

Mert A.
National Instruments

[kander1]

Hi,

Thanks for the response.  I'm pretty sure I'm cleaning things up properly, at least the threads.  I use the function:  CmtWaitForThreadPoolFunctionCompletion() to make sure the threads shut down.  Could it be anything else?

Kevin

[Mert_A.]

Hi Kevin,

Looks like Wendy has been in contact with you.  As she said, you can send your sample code to her, and she will forward it on to me.  Alternatively, you can ftp your code to ftp.ni.com/incoming (log in anonymously) and post back here to let me know what files to look for.  If you can, please post a complete, buildable project that is as simple as possible.

Also, it would be helpful to know what versions of:
1) CVI you are developing with (on the Windows machine),
2) LabVIEW RT is on the RT machine, and
3) CVI Run-Time Engine for RT is on the RT machine (check Measurement & Automation Explorer, under Remote Systems tree >> [Host] >> Software).

Thanks.

Mert A.
National Instruments

[kander1]

Hi,

Thanks for following up on this.  I sent Wendy the sample code (including the workspace and project files).  I'm using the following versions:

Labwindows/CVI 8.0.1

Labview RT 8.0

CVI runtime RT 8.0.1

Kevin

[Mert_A.]

Hi Kevin,

Your code looked good and further investigation turned up a bug!  An internal event queue is not being flushed between runs of your application.  The old, orphaned events from the first run of your app are interfering with the proper event processing of subsequent runs.  It looks like this issue applies to both the thread safe queue events and the thread pool events.

We'll be sure to fix this for a future release, but for now you can still work around the problem.  The key is to ensure that every event you generate gets processed before your application exits.  For example, in the sample code you sent, do not exit lothread's loop until you know that hithread has exited:

static int gLowthreadGo;

void CVIFUNC_C RTmain (void)
{

   ...
   gLowthreadGo = 1;
   status = CmtScheduleThreadPoolFunctionAdv(poolHandle,
                lothread, 0, THREAD_PRIORITY_LOW, 0, 0, 0, 0, &lothread_id);
   ...
   status = CmtWaitForThreadPoolFunctionCompletion(poolHandle,
                hithread_id, OPT_TP_PROCESS_EVENTS_WHILE_WAITING);
   gLowthreadGo = 0;
   status = CmtWaitForThreadPoolFunctionCompletion(poolHandle,
                lothread_id, OPT_TP_PROCESS_EVENTS_WHILE_WAITING);
   ...

}

int CVICALLBACK lothread(void *data)
{
   ...
   while (gLowthreadGo) {
   ...
}

Thank you for bringing this to our attention.  Sorry again for the inconvenience.

Mert A.
National Instruments

[deroth47]

Hi Mert --

I tried the workaround, and it is necessary to use the flags to shutdown every secondary thread, event producer as well as processor.  To keep the restart error from happening, I had to use RTIsShuttingDown() ONLY on the main thread and not on any of the secondary threads.  This seemed to fix the problem.  I will try this on our real code.  Thanks for the help and the quick response.

Don (with Kevin)

[Mert_A.]

Hi Don,

I can't see a reason why it would be necessary for producer threads to exit only after the main thread broke its loop. I can repeatedly start and stop the sample program I was sent without errors with only the modifications I posted.  The key is making sure no events get posted for a thread after that thread has stopped processing events.  In the case of an app like the sample program, in which the producer thread(s) is not registered to receive any callbacks, then it should be sufficient just to wait for all producer threads to finish before letting any of the consumer threads discontinue processing events.

If you are seeing behavior inconsistent with this and would like to follow up, I would be happy to help work it out.  Let me know if you have any problems when applying the workaround to your real code.

Mert A.
National Instruments

[deroth47]

When I just put the shutdown flag on lothread (the reader), I was able to restart the program several times (5 or more instead of just once), but it still eventually failed in the same manner, requiring a reboot.  I did this two or three times.  Putting shutdown flags on both hithread and lothread, and not using RTIsShuttingDown() except on main let me restart over 20 times before my fingers got tired (no errors).  I agree with your comment that the event producer should not need this, unless this is a problem associated with RTIsShuttingDown and the event processing.   Our final code will probably be using flags for thread control anyway, and this allows us to develop and debug the code without rebooting each time, which was our primary concern.  Unless there is an inherent problem with using global flags to exit the secondary threads, we will use this method for our code.

   Don

[kander1]

Mert A,

Thanks for your help

Kevin