Slow Unit Test Framework

[bokorpet]

Hi,

we're using LabVIEW 2012's Unit Test Framework in a project that is rapidly growing in size.

The framework does its job but it runs terribly slow. I'm not only talking about opening/editing lvtest files. To test a project with a couple of VIs of moderate complexity, 5-10 lvtest files, each having 3-5 test cases can easily take 15-20 minutes. We're using OO.

I'm afraid, the issue can escalate to a point that the usage of this test framework is infeasible for us.

Any idea how to make the framework more scalable?

Many thanks,

Peter

[Vans]

Hi Boker,

I found some hints on the reason for the slow performance. But more informations are necessary to help you.

What exactly is slow? The load process? Or the run process?

You use LabVIEW 2012. Which version of the Unit Test Framework? Real-time module? Perhaps the service pack 1 of the real-time module will help you. It includs a bug fix for ID 376884 "Project load with SCC configured is significantly slower in LabVIEW 2012 than LabVIEW 2011".

Please, feel free to come back to me.

Kind Regards,

Vanessa

AE Munich

[bokorpet]

Hi Vanessa,

thanks for your help.

Yes, it's LabVIEW 2012. No it's not real-time, it's the regular PC framework. Load and run are both slow, whch results in a long overall test time.

Maybe there is an inherent performance burden by the implementation of the test framework.  

Do you know if the test code is "interpreted" or executed as "native" LabVIEW code?

Peter 

[Vans]

Hi Peter,

could you please be a little bit more specific? I do not understand what you mean with "interpreted"? LabVIEW is a compiled language.

To improve the performance you can try to discheck the debugging feature of the VIs. You can do this in the VI properties (Ctrl + I) under categorie execution.

Can you give some numbers? How long does the execution of your VI take? How many test cases did you define? How long does it take to finish the test?

Kind Regards,

Vanessa

[bokorpet]

Ha Vanessa,

a quick interrupt in our discussion: I unchecked debugging for one of my VI under tests and it resulted in a FAILED test that was PASSED with debugging checked!

Now this scares me. What does the one have to do with the other? Do you have a clue?

The reason of the FAILED result is that LV thinks that 100% coverage is not reached (all test cases in the lvtest are PASSED). I checked and the test does have 100% coverage.

Am I missing something?

Thanks!

Peter

[Yamaeda]

Changing the Debugging will cause a recompile and affect execution times slightly, if this causes your tests to fail you have some timing sensitive construction, typically some race condition in my experience.

/Y

[bokorpet]

No, my VI under test and the test itself are totally time-deterministic.

The scenario that I described above really questions my basic trust and understanding about how the test framework works. 😞

Any hints??

Peter

[Vans]

Hi Peter,

If have never heard about this. Can you reproduce this behaviour? What if you change the settings to debugging again? Will it pass the test again? Did you change any other options in the VI properties?

Kind Regards,

Vanessa

[bokorpet]

Ok guys I've got the answer 🙂

"Code coverage is unavailable because debugging is off for the VI"

It's an N/A entry for each test case in the Code Coverage tag of the test results overview, which makes sense. Still, the test is deemed FAILED because the coverage criteria is set for the test. It's a bug (or at least a misleading feature) of the test framework as all test cases PASSED and no data on the test coverage doesn't imply FAILED!

Anyway, since our quality guidelines specify 100% coverage I cannot use the deactivated debug feature to speed up the tests..

Any other idea how to accelerate test execution?

Peter

[Vans]

Hi Peter,

good job. I will take this in mind.

Can we come back to the initial questions:

"Can you give some numbers? How long does the execution of your VI take? How many test cases did you define? How long does it take to finish the test?"

Slow performance is a relative value.

Kind Regards,

Vanessa