averaging a VEEEEERY BIG (10 MB) arrays` elements

[blue silver]

Dear Altenbach,

please see the picture above for the requested details

[smercurio_fc]

Ah, now it makes sense why you have a Scan From String function and why each number had 9 digits, which seemed odd for a 64-bit number. Your "numbers" actually represent 3 separate numbers, each being 3 characters. It obviously would have helped if you had said this in the beginning.

Suggestion: If all you want to do is to graph a running average of the numbers, you don't really need to convert the whole string into an array of numbers. Sure, you can do it, but you can also save yourself some memory by using a pointer in a loop to extract just a subset of the string you read from the file and convert just that substring to an array of numbers for which you take your running average. Since your file format is quite regular, you can easily move this pointer along the string to grab the next 10 triad of numbers. For example, in the first iteration of the loop the pointer points to the beginning of the string. You're looking to get 10 triads, so you need 10*9+10 characters (the extra 10 being the linefeed). Feed this into your Spreadsheet String to Array and Scan From String and you'll get your 10 triads. Calculate the average. Pass the average around in a shift register. Now your pointer in the string is 10*9+10+1. Repeat and rinse. There's an example that ships with LabVIEW that shows how to calculate a running average, but the idea is fairly simple.

[johnsold]

Here is a simple VI (LV 8.2) which converts the string you posted to U64 numerics. When testing, I duplicated your string 100000 times to get a string of about 2 million values. It takes about 5 seconds to convert that string to numerics.

Lynn

[blue silver]

would you pls convert to LV7.1 ?

thnx alot

[johnsold]

LV 7.1 did not have the U64 datatype (I think). But after reading smercurio's posting, some modifications are in order. Maybe I can have something for you tomorrow.

Lynn

[altenbach]

Here's one quick way to read the data into a 2D U16 array with 3 columns. (Downconverted to 7.1  from 8.0). There are many ways to do all this.

Since your data only contains 3 nibbles, adding 10 values to calculate the average will not break the U16 barrier, so you are safe here.

The problem is the averaging, because you probably need to go to floating point (do you?). Do you want a moving average of each column or of 10 successive points irrespective of column breaks? In any case, all this can be done in the same loop so you don't need to allocate any intermediary results.

Message Edited by altenbach on 11-28-2007 02:07 PM

[blue silver]

hmmmmm...

THANKS alot for your responce, BUT i have 2 problems 1

first , The data I open is like this:


DATE:12.10.2007
NAME: Mr.Merkel
ID: 10002
Country:France
Time:12.10.1907
...
...
....
100000000
1003ff001
1010013ff
1000003ff
1023fe001
102000000
1023ff3ff
.........

Here I have 2 problems,

first: I need to cut the header first and plot it in the GUI of my code, how is that possible? (cutting rows up to the first hex numbers line).
second: I have such an algorithm , if the number is over 600 , it will be converted to a "negative" number. So I need to use negative numbers in the chart , How can this be possible !

THANKS ALOT !

you are the best !!!!!!!!!

regards
silver

p.s: A code in LV 7.1 would be sooooooooo perfect un-imaginable...



Message Edited by blue silver on 11-28-2007 04:42 PM

[blue silver]

ok I will describe the whole problem simply:

1) I have a text file like this

DATE:12.10.2007
NAME: Mr.Merkel
ID: 10002
Country:France
Time:12.10.1907
Range: 10
...
...
....
100000000   - this is actually 3 hex numbers ( 3 digits long each)
1003ff001
1010013ff
1000003ff
1023fe001
102000000
1023ff3ff
.........
2 million values
..........


1) The header (up to first hex value) Should be shown in GUI.

2) After the the header info is shown, I should come up to the pure hex values , and I have to plot those values in a XY chart as converted decimals. Here the important thing is: with Range I know how many samples will be done in one second. So it can vary.

3) A moving average 10 values ( can also vary ) should be plotted separately.

Now you know the whole problem ,

PLEASE HELP !!!!

very BEST REGARDS !

silver joe

[altenbach]

---

blue silver wrote:
first: I need to cut the header first and plot it in the GUI of my code, how is that possible? (cutting rows up to the first hex numbers line).

---

If you know the number of lines to cut, just do something like in the attached modification.

---

blue silver wrote:
second: I have such an algorithm , if the number is over 600 , it will be converted to a "negative" number. So I need to use negative numbers in the chart , How can this be possible !

---

You need to be a bit more specific. What do you mean by "convert"? Should e.g. 601 be converted to -601? To -1? ...?

[blue silver]

thnx for the proposal.

I dont know how many line to cut , thats the reason I wrote, cut the header up to first hex value.

I managed to conversion part of my question. It is done in the loop, sth algortihm like 601 equals -3 , 602 to -4. solved :)))

best regards