Extract header and Data from Spreadsheet string/file

[Christian Schneider]

Hey Guys,
Here is what I am trying to do. I have a program that runs in several steps. The first section of the program does a data collection and averaging. This will output a column of values. These values can either be stored for later analysis or used immediatley. When they are stored I have set the program to convert the data into a string array and then add it to a file. (Attached is a xls of what it looks like)

What I would like to do is be able to go back later, load the file into the program and have it extract each of the pieces of information as well as the data. Somehow I need to split off the data from the string, convert that to an number array as well as pulling the other items such as the Data, time, and such.

The "&&&&" right before the data are there because I was trying something to do with a search string. They can be removed and are not an important part of the program as it stands.

The other attachement is the subvi that I use to add the header information in as well as the column headers. This files does not have column headers and they are also not needed for this particular item,

Please help me. I am running version 6i (6.0?)
Thanks
Christian

[Christian Schneider]

Oh I forgot one thing. The reason the &&&& are there was because the section labled notes does not have a specified length. Depending on the user inputs it could be either one line or multiple. This unfortunatley closes the possibility of assuming that the start of the data is that the same point all the time.

I would also  be open to changes in the method to which the information is stored.

Thanks
Christian

[Christian Schneider]

 here is an idea that I have come up with so far. It searches for the first tab and then displays whats after it. I have not yet been able to figure out how to index the rows on the string.

[Christian Schneider]

here is something else that I came up with. It seems to work but does not address the issue that the Notes section can be of varying length

[DFGray]

You set the file up well, so data extraction should not be too hard.  I always use double buffering for text files to avoid serious time issues, so the following algorithm includes this.

1. Read in 65000 bytes or the whole file, whichever is less.  This will give you your header and a lot of data.
2. Do a string search in the 65000 character string for "&&&&".  This will give you a string with your header and a string with your data.
3. Do a search in the header string for "Notes:".  Remove the whitespace from the beginning and end of the string after the match and it will give you your notes.
   
4. The string before the match is your header.  You can start processing it by using the spreadsheet string to array VI to separate out the fields.
5. The data string you got after the first match will be some or all of your data, depending on whether or not you got it all in memory.  If all is in memory, you can use the same spreadsheet string to array VI to parse it.  If not, find the last carriage return (CR) (so you make sure you have a full field) and use the data before that in the spreadsheet string to array VI.  Read another 65000 characters (or to EOF), append to the bit left from the previous after the last CR, repeat...

Check out the PNG attached for a code fragment written in 7.0.  I am not sure the icons remained the same or if 6i has the trim whitespace VI, but the general idea is the same.

[Christian Schneider]

DFGrey,

Thanks for the information. Couple questions though, would I have to make some kind of case structure or would I have to set it up to look for each of the parts of header information. From what I understand of the picture you posted it seperates data and notes out and then leaves the main portions of the header as an array.

So I would then have to go and seperate the array to store each part as a Global? This information is more than just the header, I need it to run claculations on the data so it cant be left as a group.

As of right now I ended up going with what I posted as Possibilty 2 where is seperates it by line. it works likes its supossed to but  I will try to adapt this to mine since I think its probably better since its searching rather than assuming everything is in the right place.

Thanks
Christian

[Christian Schneider]

Since I need to use the information for more calculations, what I have done was to take all the variables back out and store them to the globals which they came from, run the calculations and then store them again to the new file. Is this ok?

My program kind of goes like this:

1. Collect Data
2. Save Data with header information
3. Decide whether to analyze data now or later

4. Load data into same program if analyzing later
5. Load header to globals
6. Process data.
7. Write header to new data file.

Working on integrating you code into my program. Will post when its done.

Thanks
Christian

Message Edited by Christian Schneider on 07-27-2006 09:03 AM

[Christian Schneider]

Here is what I ended up doing. Your file made things much easier and more organized.

Still have to use the loop - Case structure to seperate the data. But it works much better.

Thanks
Christian

[DFGray]

You do have to use the loop/case structure to separate the data.  I usually use something like the attached image.  It is more robust to changes in header order, allows you to add new things to the header (don't forget a default case which does nothing to handle things not expected), and allows you to initialize your data to handle the case where the data is not present.

Two other semi-unrelated points that may make your life easier.

First, using globals to pass data around in LabVIEW is usually not a good idea.  Since LabVIEW is a highly parallel language, it opens you up to all sorts of race condition issues.  The best way to pass data around is through the use of wires.  If that is not really an option (and for large programs, it sometimes isn't), you can try functional globals (also called shift-register or LV2 globals) or single-element queues.  Unfortunately, I don't have a LV6i example.  You can find lots of info on both these methods in these forums.

Second, JPEG is a poor format choice for LV screenshots.  The lossy JPEG compression process tends to make the screenshot look fuzzy.  Try GIF or PNG.  Image quality will be better and size will probably be smaller.  If your paint program will not handle either of these, both Irfanview and GIMP are free and will do the job.  Irfanview is a viewer and easy to use.  GIMP is a full editor (with a non-Windows UI - it was written originally for Linux).

Good luck!  Let me know if you need more help.

[Christian Schneider]

DFGrey,

Thanks for the help. I will try to implemt the code that you have attached into the program.

Few things for me though. I agree Globals are not the best way to go because I have also already run into some issues where globals were over writing each other. Unfortunatley I can not change this at this point, the program is quite large, at least I think it is, and includes several distinct modules that need to be able to run indepently of each other. So far, under testing the program has not had to many issues with this and they were easily correctable.

The other problem comes from the fact that the program has to either pass data from the data collection module to the analysis or be able to load files into the memory to do analysis on those instead. That is where this whole issue with the headers comes in.

10-4 on the image processor

Thanks
Christian