how to transform data file

[wankyu]

I'm gonna make a Labview program through which one can transform a data file of 2-dimensional array form. The data file consists of (x,y) type data. First, I want to sort them in ascending order of x. Let's assmue xmin and xmax as the minimum and maximum values of x. I divide the x-range, xmax-xmin, into n intervals of equal length. And then, I want to get averages (xave, yave) of each intervals. Finally, I want to build an array from these averaged data and save it as a new data file.
Thanks in advance.

[mikeporter]

Could you post an example of your input datafile?

Mike...

[Evan]

Wankyu,
There seem to be 3 steps to your application.
1. Sort the array
2. Find the averages and build the results into an array
3. Save the data to a file.

Here is a link to an example program that you can use for step 1.
http://venus.ni.com/stage/we/niepd_web_display.DISPLAY_EPD4?p_guid=B45EACE3EA4356A4E034080020E74861&p_node=DZ52026&p_submitted=N&p_rank=&p_answer=&p_source=Internal

For step 2. First I would index the sorted 2-D array into 2 1-D arrays, representing each column. Then use a For Loop that would iterate N times. Inside the for loop use the Array Subset VI to get the intervals of data, for each 1-D array. The VI's inputs are index and length. The index would just be the current iteration count multiplied by n and the length would be yo
ur value n. Depending on the flavor of LabVIEW you have, you may have the Mean VI. If so you can use it to calculate the average for each arrays subset. If you do not have the Mean VI you would need to write your own VI to find the average of an array. You can then use auto indexing on the loop to build up the 2 arrays of averages and then build them into a single 2-D array.

Lastly Step 3. To save the data to a new file you can use the Write to Spreadsheet File VI, however I do not suggest using that VI if your above application will occur in a loop, unless you want to save to a new file every time.


Evan Collier
Applications Engineer
National Instruments

[wankyu]

Hi, Evan
Thanks for your kindness. Let me explain my question in more datails. As attached, I have a data file of (x, y) form, where x is, say, temperature and y is resistance. The total number of data points in this example file is 359. As for sorting, I can do that using "Origin" program. As you see, since the data points are not evenly spapced over temperature (x), I want to divide them into intervals of equal length of x. Let's assume that the total number of intervals (that is, total number of new data points) is 50. Then, I want to divide the temperature range, from minimum to maximum, into 50 intervals of equal length. ANd then average the data points in each interval (the number of them will be different between intervals since data points are
not evenly spaced in the original file) to get a evenly spaced resistance versus temperature data file.

[wankyu]

Thanks for your kindness. Let me explain my question in more datails. As attached, I have a data file of (x, y) form, where x is, say, temperature and y is resistance. The total number of data points in this example file is 359. As for sorting, I can do that using "Origin" program. As you see, since the data points are not evenly spapced over temperature (x), I want to divide them into intervals of equal length of x. Let's assume that the total number of intervals (that is, total number of new data points) is 50. Then, I want to divide the temperature range, from minimum to maximum, into 50 intervals of equal length. ANd then average the data points in each interval (the number of them will be different between intervals since data points are not evenly
spaced in the original file) to get a evenly spaced resistance versus temperature data file.

[mikeporter]

Here try this. There are notes on the diagram about how it works.

Mike...

[wankyu]

Hi,
Thank you. I tried to run that program. It generates a inperpolated data. But, that's not exactly what I wanted to do. Let me explain of it. Typically, when I measure resistance as a function of temperature, the resulting data is not monotonic because of thermal fluctuations like as follows;

1.535 0.031567
1.526 0.031665
1.543 0.031478
1.532 0.031556
1.543 0.031468
...

And there are 50 points in one temperature interval and 5 points in the other.
So, I want to transform the original data into some averaged data. Let us assume I have 1000 data points originally from 1K to 101K. If I divide the temperature range into 100 intervals and average the data in each interval, I would get 100 data points in a transformed file. T
he difficulty is that the number of data points in each interval is not a constant in practice. Please look at the attached example file consisting of 1647 points and I want to transform it into a file consisting of 100 points spaced evenly over x.

[mikeporter]

So the first column is the temperature data, and you want to group the data in the second column based on evenly spaced ranges in the 1st column data. And then do an average of each of the ranges so generated. Yes?

Mike...

[wankyu]

In fact, the new data may not be evenly spaced. Instead, they'll be arranged monotonically. The transformed data file should consist of
(T_averaged, R_averaged) pairs from each of the interavals. To make it clearer, let's assume the original data as follows.

1 1
2.6 2
3 3
4.3 4
5 5
5.5 3
6 6

I want to divide the x-range into two of equal length, from 1 to 3.5, and from 3.5 to 6. If I average each interval, the new data points will be
{(1+2.6+3)/3, (1+2+3)/3}={2.2, 2}
{(4.3+5+5.5+6)/4, (4+5+3+6)/4}={5.2, 4.5}
Hope this makes it clearer to you.

[Evan]

I think I understand the issue, you will have a sorted 2-d array and you want to average certain ranges of that data, however the amount of points in each range varies.

I would suggest using the Threshold 1-D Array VI in a loop. After you have sorted the array, you can calculate the first range you want, i.e. 0-n. Pass the n value to the Threshold VI and it will tell you the index of that value, whether there is a true data point there or not. The array subset below that value (0 to the output index) would represent the data set you want, and you can average that range.

Then the next time the loop iterates pass 2n to the threshold VI, This time the array subset you want it from the first threshold index found to this new one. Keep looping
until you have selected all the ranges and averaged them.