multithread DLL cleanup

[JGS]

Hello all,

I'm building a small DLL (to be used from matlab) for some data-acquisition. One of the functions can take several hours: streaming more than a million samples to a file. So for this particular function, I want (read: I am trying) to execute it as a background operation in a separate thread. After reading a NI quick-guide on multi-threading I selected the following actions:

- Start streaming function

* starts a new async-timer that calls each second ProcessSystemEvents so that any posted CVI callbacks are executed

* start new threadpool for maximum 1 thread and highest (but not time critical) priority

* schedule streaming-function for this pool (again with highest but not time-critical priority)

   with specification of clean-up function to be executed in the current thread after finishing the streaming function

- async-timer callback executed ProcessSystemEvents each second so that posted CVI callbacks are executed

- clean-up function

* Release thread pool function ID

* Discard thread pool

* Discard async-timer

- Status function that can be used from the prime thread (still available at matlab prompt) to check the status of the streaming  

The start of streaming (simulated with a counter countdown) goes well

I can retrieve the status that goes from kCmtThreadFunctionExecuting to kCmtThreadFunctionComplete.

However the clean-up function - as specified in the scheduling function - is never started.

Q1: what did I wrong that prevents the clean-up to be scheduled ?

Q2: are there different approaches/examples that do just want I want ?

Thanks in advance, Jos 

Using:   Labwindows/CVI 8.0.1

/* some global housekeeping variables */ 

static int PoolHandle = -1;
static int ThreadHandle = -1;
static int ThreadSamples = 60;
static int ThreadStatus = 0;
static int ThreadTimer = -1;
/* clean up function */

void CVICALLBACK FBGLAB_ThreadCleanup (int poolHandle, int functionID, unsigned int event, int value, void *callbackData)

{
 int Counter = 0;
 int Status = -1;
 
 Delay(0.2); /* short wait so thread function can close */
 printf("Starting pool clean up\n");
 CmtGetThreadPoolFunctionAttribute (PoolHandle, ThreadHandle, ATTR_TP_FUNCTION_EXECUTION_STATUS, &Status);
 while ((Status < kCmtThreadFunctionComplete) && (Counter < 30))
 {
  Counter += 1;
  Delay(1.0);
 }
 if (Status < kCmtThreadFunctionComplete)
 {
  /* terminate thread after timeout */
  CmtTerminateThreadPoolThread (PoolHandle, ThreadHandle, 0);
  Delay(1.0);
 }
 CmtReleaseThreadPoolFunctionID (PoolHandle, ThreadHandle);
 CmtDiscardThreadPool (PoolHandle);
 DiscardAsyncTimer (ThreadTimer);
 ThreadHandle = -1;
 ThreadTimer = -1;
 PoolHandle = -1;
}
/* simulated streaming */

int CVICALLBACK FBGLAB_ThreadStreamingWavelength(void *functionData)
{
 ThreadStatus = ThreadSamples;
 while (ThreadStatus > 0)
 {
  Delay(1.0);
  ThreadStatus -= 1;
 }
 ThreadStatus = 0;
 return(0);
}
/* async-timer callback */

int CVICALLBACK FBGLAB_Timer(int reserved, int timerId, int event, void *callbackData, int eventData1, int eventData2)
{
 if (event == EVENT_TIMER_TICK)
 {
  printf("Thread status is %0d\n", ThreadStatus);
  ProcessSystemEvents();
 }
 return(0);
}
/* part of function for retrieving streaming status */

long __stdcall FBGLAB_GetStreamingStatus(long lDevice, char *Reply, long BufferSize)
{

 int Status, FunctionStatus;
 Status = CmtGetThreadPoolFunctionAttribute (PoolHandle, ThreadHandle,
            ATTR_TP_FUNCTION_EXECUTION_STATUS,
            &FunctionStatus);
}

/* start simulated streaming */

long __stdcall FBGLAB_StartTest(long Nsamples, double SampleInterval)
{
 int Status = 0;
 if (PoolHandle >= 0) return(FBGLAB_ERROR_MULTIPOOLEXIST);
 if (ThreadHandle >= 0) return(FBGLAB_ERROR_MULTITHREADEXIST);
 
 if (Status >= 0)
  ThreadTimer = NewAsyncTimer (1.0, -1, 1, FBGLAB_Timer, 0);
 if (Status >= 0)
  Status = CmtNewThreadPool (1, &PoolHandle);
 if (Status >= 0)
  Status = CmtSetThreadPoolAttribute (PoolHandle, ATTR_TP_THREAD_PRIORITY,THREAD_PRIORITY_HIGHEST);
 if (Status >= 0)
  Status = CmtScheduleThreadPoolFunctionAdv (PoolHandle,
           FBGLAB_ThreadStreamingWavelength, NULL,
           THREAD_PRIORITY_HIGHEST,
           FBGLAB_ThreadCleanup,
           EVENT_TP_THREAD_FUNCTION_END, NULL,
           CmtGetCurrentThreadID(),
           &ThreadHandle);
 return((long)Status);
}

[JGS]

I have some additional information after some puzzling.

It seems that CmtScheduleThreadPoolFunctionAdv puts the Cleanup function in a queue but the function is not executed.

I had expected this as there is no user-interface that scans for events. That's why I put the async-timer in it with ProcessSystemEvents. However this does not to work. From a debug-printf in this timer, I know that it runs. But the ProcessSystemEvents does not fire the Cleanup function. I do not know why.

I assume that the Cleanup function is put in a queue. When I add ProcessSystemEvents to my function GetStreamingStatus (called from matlab prompt ie main thread) the Cleanup function is executed. Cleanup is not executed while the streaming is going on; Cleanup is not automatically executed after streaming is completed; Cleanup is executed with a call to GetStreamingStatus (with ProcessSystemEvents) after streaming has completed. So I assmue that the Cleanup function is put on the queue after completion of the trehad-function and that it needs ProcessSystemEvents to start. However, this is after external interaction where this set of functions should be able to do it all self. 

To solve the puzzle, I guessed that the async-timer fires a ProcessSystemEvents in its own thread and not in the main thread. So I changed the async-timer and use PostDeferredCallToThread to fire a new function in the StartTest-thread (CmtGetCurrentThreadID); this new function calls ProcessSystemEvents. No succes.

Another try that I did was using PostDeferredCallToThread or PostDelayedCall just before the return(0) in the streaming function. With either of these, I tried to start a new function that calls ProcessSystemEvents with a second delay and therefore starting the Cleanup.

Q: How to get the Cleanup started - in the original thread pool - in a DLL where there is no user-interface that scans for events ?

[dummy_decoy]

concerning the cleanup function: i am not sure, but if your thread pool is limited to only 1 thread, the cleanup function may have to wait for the streaming function to complete before being executed. while the cleanup function is a callback called from the thread termination, i would still try by increasing a bit the number of threads of the pool. anyway, most of the code of the cleanup function seem useless: if the callback is called the thread is already completed, so you should not have to query its status. also, why not include the cleanup into the streaming function  or at the end of a wrapper around the streaming function ? this way, you are sure it will be called at the end of the streaming.

also, the status function does not seem right: if your callback is called, then the thread handle has been erased in the cleanup, so you should not be able to retrieve the status of this thread as you do not have its handle anymore.

there is still something bugging me: why does your streaming function need an event queue ? only for the thread pool scheduling ? does it updates a GUI ? i do have the feeling that your design is a bit wrong. personnally, i would either:

- let the streaming function run for hours and let the user of the DLL manage the multithreading part. it is the simplest approach. while people may think that it's just being lazy, it has the advantage of letting the user choose the way the function is scheduled and prioritized.

- provide a completely asynchronous API consisting of a start/stop function and a completion/status function or callback.  this could be implemented in a simple way using a single call to the CreateThread() Windows SDK function (i am sorry NI, i still don't buy the thread pool idea: when i need a single thread, i don't need all the overhead of a pool and scheduling and callbacks and ...) and it should not need an event queue.

i hope this helps...

[JGS]

Thanks for the ideas ..... but it does not help me yet

I have increased the number of threads in the new pool to 2, but this does make no difference.
You are correct that some of the code in the cleanup should be superfluous. But as my first attempt did not work I added some safety/delay/checks in the cleanup. It will be removed as soon as the code works.
 
You are also right with respect to the status function. The actual function on my computer has 10 lines that checks/reports the existance of a valid pool and thread handle. And it has another 10 lines that translates to results to readable message. Because my message became long, I showed only the single status-checking function.
 
Now to the core of the problem: does the software need an event queue ? Or why do I need ProcessSystemEvents ?
None of the other functions in the DLL needs an event queue. But i) I want to clean up the thread pool and ii) CmtScheduleThreadPoolFunctionAdv has a special parameter for a callback function. When it did not work as advertised I  thought about the differences between my DLL and common GUI applications. And one of the possibilities is that the function is invoked when (common with a GUI) CVI processes events in the main thread (like PostDeferredCall). So I tried to force this with ProcessSystemEvents. I think I succeeded partly as explained in the first follow-up. But now it needs to be done automatically. And this I have not succeeded yet.

I do not agree with you on letting the user decide or take action. When I build such a tool, all overhead activity has to be invisible to the end-user. And selecting yes/no a separate thread for this/all functions is not on the list of wishes for the intended users.
With respect to the Windows SDK functions, I am not familiar with these. I picked the NI pool/thread model because it is explained how to implrement multithread with many examples in Labwindows. But if I get no further ideas, I am (almost) stuck and will consider the Windows SDK.
I still hope that someone knows why/how to get the Cleanup fired. As an alternative, I can create the pool in DllMain on attach, and I can destroy the pool in DllMain on detach. But than: I do not know if detach always occurs, even when the main application (matlab) just exits without unloading the library.

 Thanks anyway, Jos
 

[dummy_decoy]

i am sorry, i presumed a lot about your code without having the whole software.

i am not sure (since i generally don't use thread pools), but scheduling a function into a thread may be using the event queue.someone at NI should tell us more regarding thread pools.

did you try the wrapper ? write a function which calls the streaming function then perform the cleanup, schedule this function into the thread pool and remove the callback.

regarding your users: that's right, letting the user do the multithreading involves having an advanced programmer as a user. that's why i the asynchronous interface, but it may not fit correctly with matlab.

regarding the Windows SDK threading functions: they are not difficult to use, and there is some examples in the SDK but the are difficult to find with the missing table of content in the version that ships with CVI. first try the wrapper function. if you decide to go the Widows SDK route, tell us, and i will provide you with example code and links to documentation.

[JGS]

Hello,

I am still hoping that a NI specialist - or a forum user with similar problems in the past - gives an answer. Why the Cleanup is not executed and/or how to get it running. If this does not happen, I will try to create and cleanup the thread pool in DllMain.

With respect to the wrapper, I did not try this idea. Maybe I am assuming to much, but .... I do not think that I can remove the threadpool while a function is running in this thread. So the wrapper has to be outside the thread.

With the wrapper in the main thread and with the requirement that the application runs in the background, the matlab-call or the wrapper has to launch a PostDeferredCall; the function has to return so that it can continue and exit to the matlab prompt. The manual for PostDeferredCall explicit mentions that it is executed while the main thread processes events. So I need the  (same ?) ProcessSystemEvents as I need now.

I am still thinking about a wrapper that schedules the acquistion, that waits 1 second so the acquisition has proabably started, and than schedule a second function in the main thread. If this is possible and if this his second function can wait at low processing power, it can wait until the acquisition has finished. But the easy examples become more complex with each new idea.

I am starting to think that the thread pool is not as easy as it seems. Due to the my problem, I am starting to think about the handles: why a thread handle and not a threadpool handle ? How to send a function call to - any - thread in the mainpool ? Is the DLL running in the  "same thread" and can mainthread=CmtGetCurrentThreadID be used all the time ? When I do not find a white-paper on this kind of information, I certainly will look into the Windows SDK possibilities.

Thanks anyway for the ideas and the offer.

[menchar]

It's sometimes just an issue of personal preference, but I've always preferred using the Win32 API directly for multi-threading issues.  It's often hard enough to understand what the Win32 OS is going to do in re threading, without also understanding what NI has implemented.  While NI's goal is fine (provide a non-system-dependent multi-threading interface), this stuff can be so exacting as to require an absolutely sound understanding of nuanced thread behavior.  And when you think of it, NI can't possibly have implemented threading functions that aren't direct uses of the underlying Win32 threading capability or aren't sythesized from it (assuming of course you're running on Win32).  Beveridge & Weiner ISBN 0-2-1-442342 is getting a bit dated but remains an excellent resource for gaining a sound understanding of multi-threading in Win32 OS's.

Win32 OS's schedule threads, not functions - a thread starts in the "thread function" but then it goes where it goes -the OS stops and starts it as a thread, without regard to what function it's in (this is a general statement for threads running in user space).  You can implement function-specific threading behavior in your code, by using kernel synchronization objects (mutexes, semaphores, events, signals), as well as critical sections, Sleep() calls, etc., but the OS isn't going to do it unless you design your code that way.  And the OS is going to try and run everything at "normal" thread priority unless you actively set thread priorities.  Even if you don't, the OS will constantly adjust thread priorites and time slices (quanta in MS lingo) according to it's own idea of what should happen.  Relying on thread priorities for specific thread scheduling behavior rather than some discrete mechanism (e.g. WaitForEvent) can be a bad idea - you'll think you have it set and then the OS goes and diddles the thread priority a bit and things don't work.

With multi-core CPU's we now have true concurrency, and this can gum things up a bit - even experts get surprised when their multi-threaded apps that worked fine on single cores now break on a multi-core machine.

You may or may not realize that Win32 enforces specialized thread scheduling behavior whenever a thread is in a DllMain function.  I don't know that this is necessarily affecting your code, however. 

And it can sometimes be an adventure trying to figure out just what NI has implemented in some of their libraries (dll's) when it comes to multi-threading.    All are supposed to be thread safe now I do believe.

There's a lot of overhead in learning how to do a good multi-threaded application.   Sometimes you can implement a coarser, multi-process solution that yields concurrency but is a bit easier to do since you have to worry less about coordinating access to common process resources.

Good luck.

Menchar

[nickb]

Hello,

I think that the primary reason the cleaup callback is not being executed is because of where you were processing events.  As the help for the 'Callback_Thread_ID' parameter states:

The ID of the thread that you want the thread pool to use to execute the Callback Function. Pass RUN_IN_SCHEDULED_THREAD if you want the callback called in the same thread that executes the Thread Function. Pass the return value of CmtGetCurrentThreadID if you want the callback called in the thread that is calling CmtScheduleThreadPoolFunctionAdv.

If you do not pass RUN_IN_SCHEDULED_THREAD, the thread you specify in this parameter must process events using RunUserInterface, GetUserEvent, or ProcessSystemEvents.

If you pass RUN_IN_SCHEDULED_THREAD, the thread pool calls the callback directly. You do not have to configure the scheduled thread to process events in order to receive the callback.

In your CmtScheduleThreadPoolFunctionAdv, you specified CmtGetCurrentThreadID for the  Callback_Thread_ID parameter, which means that you must process events in the same thread that called CmtScheduleThreadPoolFunctionAdv.  However, previously you had been calling ProcessSystemEvents in the callback for the asynctimer, but the problem with this is that the asynctimer callback is actually run in a new thread.  To process events in the same thread as the function that called CmtScheduleThreadPoolFunctionAdv, you could modify your code as follows:

long __stdcall FBGLAB_StartTest(long Nsamples, double SampleInterval)
{
    int Status = 0;
    int flag = 1;
    int i = 0;
   
    if (PoolHandle >= 0) return(FBGLAB_ERROR_MULTIPOOLEXIST);
    if (ThreadHandle >= 0) return(FBGLAB_ERROR_MULTITHREADEXIST);
   
    if (Status >= 0)
        ThreadTimer = NewAsyncTimer (1.0, -1, 1, FBGLAB_Timer, 0);
    if (Status >= 0)
        Status = CmtNewThreadPool (1, &PoolHandle);
    if (Status >= 0)
        Status = CmtSetThreadPoolAttribute (PoolHandle, ATTR_TP_THREAD_PRIORITY,THREAD_PRIORITY_HIGHEST);
    if (Status >= 0)
        Status = CmtScheduleThreadPoolFunctionAdv (PoolHandle,
            FBGLAB_ThreadStreamingWavelength, NULL,
            THREAD_PRIORITY_HIGHEST,
            FBGLAB_ThreadCleanup,
            EVENT_TP_THREAD_FUNCTION_END, NULL,
            CmtGetCurrentThreadID(),
            &ThreadHandle);
   
    while (!done)
        ProcessSystemEvents();
   
    return((long)Status);
}

Alternatively, you could specify RUN_IN_SCHEDULED_THREAD instead of CmtGetCurrentThreadID for the Callback_Thread_ID parameter.  Then, as the help states, the thread pool would call the callback directly, with no need to process events.  Please let me know if I can help clarifying anything else!

NickB

National Instruments 

[JGS]

Hello Menchar,

Thanks for your reply. It's making me a little afraid. My regular job is R&D. For dedicated automated laboratory measurements, I do some programming in Labwindows as a side job. Often the measurement procedure is more complex than the programming itself. From the Labwindows examples and white paper, I thought is was not to difficult to get a background / high priority thread running. This is indeed easy, but I wanted also to cleanup afterwards. Now I have not yet succeeded I hesitate to look into Windows SDK. Message 5 seems quite optimistic. Now I have read your reply, and I have really serious doubts.

Hello Nick,

Thanks for your reply. The important message seems to me: I need ProcessSystemEvents (or something similar) for starting the callback. Summarizing some of the above ideas which you can confirm or correct:

- I want to run acquisition at high priority AND I want to run it in the background. For this, a threadpool is the proper tool.

- While acquisition is running in the background, I want to ga back to the matlab-prompt. For this I need to exit the initial calling function. This is no problem because the acquisition thread is scheduled and starts immediately.

- I cannot discard a thread pool while functions are running in it; especially not if the cleanup function itself is running in this threadpool.

- A DLL has not an active event handler, such as a GUI-based RunUserInterface.

Looking at your code-insert, the initial calling function keeps running while !done. So it will NOT return to the matlab-prompt until the acquisition is finished and the cleanup sets the variable to done.

Your alternative is running the cleanup in the thread that is used for the acquisition. Fine for me if you confirm that the cleanup can discard the thread that it uses itself. Seems illogical to me.

Not in the first code: I tried a PostDeferredCallToThread at the end of the acquisition. It is calling the cleanup in the initial thread. Not working. Probably for the same reason: there is no active event/function handler that fires the queued function.

So assuming I still want to cleanup the thread (release the function ID and discard the thread), I am still looking for a method that automatic fires this cleanup function. (see message 2). Or I am looking for an alternative reply: don't try because it is not easy or impossible with Labwindows.

[JGS · Accepted solution]

Hello again,

It seems that the solution is as simple as the cause.

- I create a high-priority threadpool and schedule a high priority acquisition function. Scheduling is done with CmtScheduleThreadPoolFunctionAdv. I do not use the option for a callback at the end of this threadfunction.

- A second later (or from within the acquisition function) I schedule a normal priority cleanup in DEFAULT_THREAD_POOL_HANDLE. Scheduling is done with CmtScheduleThreadPoolFunction. Inside this function I wait until the acquisition-thread is finished: CmtWaitForThreadPoolFunctionCompletion

- Because all these functions are "scheduled" the initial calling function returns. I get the matlab-prompt back for other tasks.

I have still to check which wait-statement requires least processing power e.g. CmtWaitForThreadPoolFunctionCompletion or Delay with status-check.

But I have already seen that this combination:

- starts the acquisition in a high-priority threadpool

- releases afterwards the function allocation and discards the high-priority pool

- meanwhile I got the matlab-prompt back and can excuted other task

Thanks everybody for their contribution, Jos