Reducing program's CPU footprint while maintaining speed

[Rodnebb]

I am currently creating a program that will chew through up to a Gb or two of data and push it out as excel files.

I have 2 problems:

• the conversion process takes a long time.
• I do not want to make the computer unuseable while this program runs.

So, my concern here is firstly how to reduce my CPU/memory footprint without sacrificing too much speed. I have noticed I can sometimes use near 100%CPU without disturbing other programs too much, but sometimes not.

The methods I have thought about for stopping CPU hogging is:

• insert small waiting periods into loops
• programmatically reduce the programs priority in windows.

Any other ideas?

If I am to insert waiting periods, are there any rules of thumb regarding how long they should be? Some processes seem to let go of the CPU by default, while others don't. Any difference here between development mode and a finished compiled program? And I don't want to slow my process down more than necessary.

I do not know how to programmatically reduce the priority of the program in windows - can anyone help me?

The other approach is to improve program efficiency  and speed, both CPU and memorywise. I have arrays up to 8Mb in size, which are sometimes shown on the front panel. Are there any manuals out there regarding LV's behaviour with programming suggestions? For instance I wonder what effect it has to have a 12 x 50 000 array display on the front panel of a vi the user sees, but "outside" of the window (the user can see), or the effect of passing this array through various control structures.How does use of local variables affect performance?

Thanks,

[tst]

First, your best bet is probably using a wait of 0 ms - this causes LV to check if the CPU is needed and if it isn't, it keeps working. You should definitely put waits in while loops (usually between 0 and 100 ms, depending on the functionality of the loop), but if you have long running for loops, you should put some in there is well. This should work the same in the RTE.

As for performance, data copies and dynamic resizing are the two performance killers. Local variables and UI displays cause copies, so avoid them and use shift registers instead. These will usually force LV to use the same memory area. If you're using 8.5, you can use the new in-place structure, which helps with some common bundling-unbundling situations.

Avoid resizing arrays and strings in loops. Building arrays in for loops is OK if you use autoindexing, but another way is preallocating your array before the loop, passing it in a shift register and replacing the relevant section each time. An array of 8 MB should be a breeze for LV if you don't create many copies of it.

If you want to transfer data between loops\VIs, use queues or LV2-style globals, which can operate in-place as well.

Use the profiling tools to detect memory consumption in specific VIs. You can see memory allocation points by telling LV to display buffer allocations

Lastly, it already seems that you're doing this, but just in case you're not - chunking is important when dealing with large data sets. LV can deal with a lot, but if you throw too much at it at once, you're liable to get into trouble.

Some relevant reference material:

• Search this forum for a presentation called "handling large datasets in LV", by DFGray.
• Search the site for Steve Rogers' presentation from NIWeek called "hide the dots" about detecting performance penalties and potential enhancements.
• LV comes (or at least used to come) with a PDF document about memory management and performance. If you can't find it, try searching for app note 168. It's not fully up to date, but it is still basically correct.