Splitting and type casting huge string into arrays

[JPal]

Hello,

I'm developing an application which is supposed to read measurement files. Files contain I16 and SGL data which is type casted into string. I16 data is data from analog input and SGL is from CAN-bus data in channel form. CAN and analog data is recorded using same scan rate.

For example, if we have 6 analog channels and 2 CAN channels string will be (A represents analog and C represents CAN):

A1 A2 A3 A4 A5 A6 C1 C2 A1 A2 A3 A4 A5 A6 C1 C2 A1 A2 .... and so on

Anyway, I have problems reading this data fast enough into arrays. Most obvious solution to me was to use shift registers and split string in for loop. I created a for loop with two inner for loops. Number of scans to read from string is wired to N terminal of the outermost loop. Shift register is initialized with string read from file.

First of the inner loops reads analog input data. Number of analog channels is wired to its N terminal. It's using split string to read 2 bytes at a time and then type casts data to I16. Rest of the string is wired to shift register. When every I16 channel from scan is read, rest of the string is passed to shift register of the second for loop.

Second loop is for reading CAN channels. It's similar to first loop except data is read 4 bytes at a time and type casted to SGL. When every CAN channel from scan is read, rest of the string is passed to shift register of the outermost loop. Outputs of type cast functions are tunneled out of loops to produce 2D arrays.

This way reading about 500 KB of data can take for example tens of seconds depending on PC and number of channels. That's way too long as we want to read several megabytes at a time.

I created also an example with one inner loop and all data is type casted to I16. That is extremely fast when compared to two inner loops!

Then I also made a test with two inner loops where I replaced shift register and split string with string subset. That's also faster than two inner loops + shift register, but still not fast enough. Any improvement ideas are highly appreciated. Shift register example attached (LV 7.1)

Thanks in advance,

Jakke Palonen

[JPal]

And here is the string subset example.

[altenbach]

There are many possible ways to slice and dice this. 🙂

For example, if the number of I16 and SGL channels is fixed, you could cast it into an array of clusters with the correct elements. If not, you could cast it to a I16 array, reshape it to a "x+y*2 by N" 2d array, split it and recast the SGL part to SGL.

The attached example implements the two solutions discussed, see what's faster. 🙂 Good luck!

Message Edited by altenbach on 08-05-2005 09:08 AM

[Ray.R]

Good suggestions Altenbach,

I vote for the cluster approach because it is clean. 

Both approaches should be comparible in speed....  hummm... I'm curious...

Hopefully we get feedback from your suggestions  😉

JLV

[altenbach]

OK, there is clearly room for improvement. I did some timing and my two above suggestions are already about 100x faster than yours. A few teeaks led to a version that is now over 500x faster than the original code.

A few timings on my rather slow computer (1GHz PIII Laptop) are shown on the front panel. For example with 10000 scans (~160kB data as 6+2) my new fastest version (Reshape II) takes 14 ms versus the original 7200ms! It can do 100000 scans (1.6MB data) in under 200 ms and 1000000 scans (15MB data) in under 2 seconds. 😮

I am sure the code could be further improved. I recommend the Reshape II algoritm. It is fastest and can deal with variable channel counts. Modify as needed. 🙂

Attached is a LabVIEW 7.1 version of the benchmarking code, containing all algorithms. I have verified that all algorithms produce the same result (with the limitation that the cluster version only works for 6*I16+2*SGL data, of course). Remember that the reshape function is extremely efficient, because it does not move the data in memory. I have some ideas for further improvements, but this should get you going. 🙂

Message Edited by altenbach on 08-05-2005 03:06 PM

[JPal]

GREAT thanks!!!

Channel number is not fixed so using cluster method is bit difficult, but you have provided plenty of solutions to choose from 🙂

I also tried using string subset and split string in for loop to cut every "scan" in I16 and SGL parts. Then type cast both to 1D arrays -> tunnel with indexing -> 2D arrays. That was much faster than the examples I sent, but still time was nothing near compared to your benchmark results (and I was using 2.4 GHz P4) .

I'll look at your solutions and compare results tomorrow when I'm at my project.

[JPal]

Thanks altenbach, your code is working just great! 🙂

I did some comparisons using my string subset -> split string -> type cast to 1D array method and your reshape2. Reshape2 is 3 - 4 times faster than my code. I could convert 28 megabytes of data in 734 ms with reshape2. Same data took 2734 ms with my method. Thanks a lot again, my project is looking much better now

[Ray.R]

Nice panel Altenbach,

I like it.  😄

[altenbach]

Thanks! 🙂 It's probably possible to tweak a bit here and there. However, I don't think there is much slack left, especially since you need variable channel counts. Maybe somebody else finds a faster solution? 😉

One ugly bit on the current diagram is the fact that when I dropped the table on the FP, I did not notice that LabVIEW (in its infinite wisdom! ;)) dropped the table terminal inside the FOR loop for the data generation. This should of course be outside any structures, at least for cosmetic reasons.