Benchmarking VI (getting offset each time when benchmarking)

[WG-]

Hello

I've created a benchmark VI which can benchmark numerous VI's. The following rules however apply.

- The benchmarked VI's must have the name <name>(i).vi where i = {1,2,3...n}

- The benchmarked VI's must have the same connector wiring. 

Now the problem that I'm having is that after each benchmark a "offset" in the time is being created. Each benchmark this offset is getting larger. Now I want to know where this comes from. One thing I exclude for myself is that it comes from the benchmarked VI's because they don't include any loops.

To be noted is that when I however shutdown my benchmark VI and start it again the offset is gone.

So any one sees any mistakes, problems?

Here is a VI snippet screenshot (download the zip, to test it, it contains the benchmarked VI's.)

[altenbach · Accepted solution]

First I was confused what you mean by "offset"? Especially since your outputs are also called "offset ..."... 😉

I agree there is something terribly wrong at first singht. On my computer, the short tests complete in about 0ms, but in the next run they complete in 75ms, the next run 150ms, and so on.

The problem is that in "generate strings" you don't initialize the shift registers!

You only reset in "first run?", instead of every press of the benchmark button.

[RavensFan]

Nice job on creating a framework for testing different VI's.

Can you post a screenshot of what your VI's front panel looks like when your test is done?  I don't understand what you mean by an "offset" in the time.  When I run it, I get a pretty linear relationship between number of elements and time to run.

If I can recommend a couple changes to your program.  The 20 iterations can take a long time to run.  Although you have several progress bars on the screen, it's not clear what each means, and with the larger sample sets, there are long periods of time where it looks like nothing is happening.  I would add a progress bar in some manner that lets you know it is in the middle of a long calculation sequence.  Of course this could also upset the timing as any screen updating in the middle of a calculation would affect the benchmark time.  Another alternative is to have an upper limit run time where if a subVI takes longer than some user specified run time, then it doesn't move on to the next larger iteration.

Another thing might be to update the graph between each subVI and each iteration so you can see the progress of the lines as the VI runs.

[altenbach]

---

altenbach wrote:

You only reset in "first run?", instead of every press of the benchmark button.

---

In particular, look at the array sizes coming our of your "generate strings" subVI.

1. iteration: 2,4,8,16,...

2. iteration: 1048578, 1048580, 1048584, 1048592, ...

2. iteration: 2097154, 2097156, ...

You are carrying a lot more baggage with each benchmark...

[altenbach]

Some general comments:

• remove the innermost progress bar and move the stop benchmark button one loop out. You don't want to benchmark front panel interactions in the innermost loop, it skews your timings.
• disable debugging
• Setting the "generate data" to false needs to be moved one loop out. It needs to be done only once and not for each subVI. It's the only reason you might want to retain the outer flat sequence structure, which is currently useless.
• The "iterations" calculation is too complicated. Try to use "scale by power of 2" based on [i].
• Use "format into string" to generate the file names. No concatenation needed.
• ...

[WG-]

@Altenbach

YES that was the problem!!! I totally overlooked that. 

To be clear...

The problem was that when I benchmark the first time; the first iteration was 0 ms. When I then started a second benchmark; the first iteration was 75ms and so on, as Altenbach says.

@Ravens Fan,

"The 20 iterations can take a long time to run",

that number can be changed of course.

"Although you have several progress bars on the screen, it's not clear what each means",

I will try to make that more understandable. But in general. The first progress bar is the progress of the iteration loop of 2^i samples (i is a number between 2 and j). The second progress bar stands for the number of VI's. The third progress bar stands for the total of iterations that has to be done, i.

"and with the larger sample sets, there are long periods of time where it looks like nothing is happening." therefore is the "Generating data" Boolean. When it is turned on, it means of course that it is generating samples.

Further I will see what I do with your comment, I just made this VI for a little fun because I really wanted to know which strpos VI version I created was faster 😛

[WG-]

remove the innermost progress bar and move the stop benchmark button one loop out. You don't want to benchmark front panel iterations in the innermost loop, it skews your timings.

Ya I was thinking about that, but I'm not sure if I will. Since the case is, is that I want to know which solution is faster not "how fast it is". And since each VI is "disturbed" by this it won't make the results incorrect.

disable debugging

Ok, but where can that be done?

Setting the "generate data" to false needs to be moved one loop out. It needs to be done only once and not for each subVI. It's the only reason you might want to retain the outer flat sequence structure, which is currently useless.

No can do. The generate data VI generates the data and only adds the missing data 2^(i-j) each iteration.

The "iterations" calculation is too complicated. Try to use "scale by power of 2" based on [i].

Ya your right that is the same as I'm doing now, I will rewrite that. I forgot that there the "x*2^n" VI.

Use "format into string" to generate the file names. No concatenation needed.

Ok

[altenbach]

---

@WouterG wrote:

Setting the "generate data" to false needs to be moved one loop out. It needs to be done only once and not for each subVI. It's the only reason you might want to retain the outer flat sequence structure, which is currently useless.

No can do. The generate data VI generates the data and only adds the missing data 2^(i-j) each iteration.

---

Sorry, I meant writing to the "generating data" boolean indicator local variable. You are hammering it with "false" numerous times while it is already false from the first iteration. Right?

[altenbach]

Here are a few simple alternatives (using LabVIEW 2010, so the locals and boolean constants look a bit different ;))

It is safer generating the file names with list folder and a appropriate pattern. Now you are guaranteed that all files exists. (If you generate by the numbers and one is missing, it would cause trouble).

Top: using formatting. Bottom: using list folder.

.

Here's one way to calculate the iterations. As you can see, the boolean is set to false only once, delayed by dataflow.

[WG-]

You shouldn't be sorry, I should be sorry, I just see that I readed your comment wrong... 

Thanks for the example with the sequence (now the big one can be removed indeed) and with the list files, never really worked with that to be honest, but I'm sure to look into it now!

I looked for setting off debugging, I guess you meant the setting off the option "Enable automatic error handling" in the VI properties -> Execution?

You shouldn't be sorry, I should be sorry, I just see that I readed your comment wrong... 

Thanks for the example with the sequence (now the big one can be removed indeed) and with the list files, never really worked with that to be honest, but I'm sure to look into it now!

I looked for setting off debugging, I guess you meant the setting off the option "Enable automatic error handling" in the VI properties -> Execution?