slow conversion of base64 to sgl array and back

[altenbach]

OK, I quickly implemented a simplistic "SGL Array to Base64" and "Base64 to SGL array". Note that I don't deal with padding. For example your input string is one character too long, which I deleted. Now both directions give the same result as yours.

Note that I don't use any boolean arrays. As we have learned in the bit twiddling challenge, These are typically slow.

I am sure things could be optimized further.... Not sure how it compares with Lynn's version...

[johnsold]

Nice.

I dropped altenbach's VIs into the same test VI I have been using and ran it against the profiler in the same manner as the things I tried. It runs about 23700 us.  This compares to 8300 us for the original and 1670 us for my fastest under the same method. The Bas64ToSGLArray.vi takes about twice as long as SGLArrayToBase64.vi.

Lynn

[altenbach]

So you are saying mine is orders of magnitude slower? 😮

How are you testying? If I test mine on the default data, pack and unpack time are about 1ms, while your takes about 500ms on this computer. <Shrug...>

[johnsold]

I put your subVIs in the test VI the OP posted and removed everything else. I run the Profiler.  All subVIs have panels closed.

Here is the file saved by the Profiler for one run each.  mzml data unpacker v1.vi is the original. v.2a uses the altenbach subVIs. v.1.7 uses my subVIs.

Lynn

[altenbach]

Well, I still don't understand how you are testing. Using the profiler seems unsuitable. What if you put your subVIs (after a few quick adaptations) into my test VI and measure the elapsed time?

As I suspected, my VIs still had quite a bit of slack left in them. It turns out that the parallelization of the loop does not gain anything. The parallelization overhead seems to cost more than it gains. I got rid a a few more allocation dots and did some general cleanup and streamlining and they are now about 5x faster than before (150-300us for the default test data).

(My times are under LabVIEW 2012 32bit and Windows 8 64 bit, running on an old Intel Q9300 quad core)

(I did experience some weirdness in benchmarking in the previous version where everything got suddenly about 200x slower (200ms!). I had tons of other VIs open. After restarting LabVIEW, things got fast again. Maybe the memory was way too fragmented? Not sure what happened).

[altenbach]

Using data from 7M random SGL points as in the original problem description (~37MB of Base64 encoded input data!), my times are ~12s to decode and 6s to encode. How does yours compare?

I am sure things could be sped up even more... I have some ideas but probably no time to try them. 😉

[dpak]

Dear Altenbach and Lynn,

Thank you both for your advice and help.

Atenbach - re the string length - yes, I just copied and pasted the first part of the string, paying no heed to the precise character length.  Padding is something I will incorporate later.  Luckily, in the input files, that is normally taken care of by the programme that generates those files.

I will have to take some time to understand Altenbach's method - very fast but fiendishly advanced compared to my normal use of labview (15 years in and I'm very aware how little I know!).

I will compare both methods on our actual runtime machine - I'm just looking at them on my laptop just now.  On this machine, as a quick test, Altenbach's method took 17s to convert both the mass axis and the intensity axis of a test file.

I still have a Labview theory question - why is typecasting slower than my clunky method of dealing with the exponent and mantissa sub-arrays "arithmetically"?  I had assumed that typecasting would be very fast, but does it do something unusual in the background?

David

[johnsold]

I took your test VI and placed my code inside. I made one other change - I removed the progress slider to be consistent with your code.

When testing I noticed that occasionally your code executes much faster than most other times. So I wrapped your test code in a for loop and calculated minimum, maximum and mean times.  LV 12 SP1. Mac OS X.

My mean times are faster than yours, but occasionally your minimum is much lower than the minimum on mine. (1000 iterations).

     

These tests are with the default string (from which the "+" was removed).  I have not run them with large datasets.

Test Base64ToSGLArrayandBack.loop.vi has altenbach's code. The version with .L.Loop in the name has my code.

Lynn

[altenbach]

What is your processor?

[altenbach]

Well, not fair, you did not use my V2 posted above. As I said, the parallelization is not worth it and there were too many data copies. I think that also causes the variability in results.

Here are your benchmarks after substituting V2 subVIs. 😉

 

Seems I am about 4-8x faster on my PC. 😄