Newbie Guidance: Data Collection, Storage, Review - Best Method

[jqm]

Hello,

As the title states, I am a relative newbie to labview.
I am using labview to gather data from instrumentation such as the National Fieldpoint Strain gage measurement SG-140, RTD-124, as well as an agilent 34970A data acquistion switch unit. I am typically monitoring such variables as voltage, resistance, temperature, and time.

In the all of my custom apps so far I have been using LVM and converting to excel manually. This is time consuming.

In the long term I would like to create applications for manufacturing;

for example when manufacturing load cells, they have to go through a temperature compensation process where the load cell output is monitored over a temperature range, and any changes are compensated for. In order to compensate for any thermal effects I would need to know output voltage (when the cell is considered "stable" maybe through a moving average) and at an elevated temperature, this value can be stored, along with a timestamp. This has to be repeated for another temperature...from this info temperature compensation can be calculated and added to the load cell.

So I think think that I would have to perform post processing on a data set, that could look at the data, perform the calculation, and then present that to the operator.

I hope that I am making sense.
Any comments or suggestions are greatly appreciated.

Thanks
jqm

[CoastalMaineBird]

Well, I understood what you said, but not what you are asking.

I have had many situations where I post-process data.

Please specify exactly what you're looking for.

[jqm]

Sorry I did not present the question clearly, I guess being a newbie makes it difficult to know what to ask for.

but knowing my application;

1) How would you store data; text file, database, other?
2) how would you go about analizing the data and presenting it to the user?

there are arrarys, lvm files, databases; labsql, labview database toolkit, access, etc.


It looks as if there are a couple dozen ways of storing and looking at data. I am curious as to what the best approach ( or most common approach) would be as I dont have time to try all methods.


Thank you for your time

[CoastalMaineBird]

1) How would you store data; text file, database, other?


There's no way to answer that definitively without knowing what you're going to do with it.

If you're going to analyze it later in LabVIEW, then use a datalog file. Stay away from text files if you can.

With a datalog file, you create a cluster containing whatever you want to store - however complicated you need to make it.
When it comes time to write it, you use three functions: NEW FILE, WRITE FILE, CLOSE FILE. Period.
When it comes time to read it, you use three functions: OPEN FILE, READ FILE, CLOSE FILE. Period.
Another advantage is tamper-resistance. If the user opens up this file in NOTEPAD, he will see garbage, and not be likely to try to edit it. You don't have to range-check every value in the file because some user might have changed the NUMBER OF CHANNELS field from 6 to 60.
Another advantage is in file dialogs - if you tell the dialog to show files with a certain datalog type, then that's all you see. Regardless of extension, the user can't choose a file that's in the wrong format.

However, if you have to be compatible with some other (non-LabVIEW) program, then it's not the right choice. I suppose you could make a C program to parse a datalog file and extract the right stuff, but it's a lot of work. But maybe you HAVE to share the data with a spreadsheet user. Then write it out in spreadsheets. I have written programs that will save and load files in datalog format, but will export (not import) spreadsheet files. The best of both worlds.

Use a database if you need to query it later with things like: "Show me all the runs where the Cell temperature exceeded 50 degrees C" or some query like that. You COULD write a LabVIEW program to extract those, but databases are already there for that.

2) how would you go about analizing the data and presenting it to the user?

Again, that's too general for me to answer. Take a look at my Portfolio, particularly the first two items (looks like I need to update that). Those are LabVIEW programs designed to analyze and display data.

Your first question has to be: "What does the user want to see from this data?"
Is it graphs? Statistics? Pass/Fail judgements? Printed reports? All of the above?

Start there - Design the final page(s) that your users see and do it first. Make sure you include all options - if he wants to click a button to see a graph, then find that out and include it in your thinking. That's your bottom line. Everything else goes toward filling in those blanks.

It looks as if there are a couple dozen ways of storing and looking at data. I am curious as to what the best approach ( or most common approach) would be

Well, the reason there are a couple dozen ways, is that there is no one "best approach" to all problems. As I said, start ant the end, and work backwards.

[Dennis_Knutson]

Binary data storage is the fastest way to read/write and creates the smallest files. If you are doing high speed data acquisition and need to stream to disk, binary storage is usually the only way you can keep up. The downside is that the data is not easily readable. You can't open the files in Notepad or Excel. To read them, you have to know the exact method used to write the data (i.e. as U8, U16, SGL, DBL, etc.). Text files take longer to read and write but can be easily viewed by other applications. Saved as comma or tab delimited, they can be imported by all spreadsheet programs. Files saved as proprietary formats such as Excel, allow you take advantage of the special features of the program. NI sells the Report Generation Toolkit to make writing directly to an xls file much simpler. You can do it yourself with Excel's ActiveX methods and properties. Binary, text, and Excel files require you to come up with some method of file naming and storage. You might want to name each day's file differently by date, each run by date and time or by serial number, etc. Managing data stored in multiple files accumulated over days, weeks, years, can be a problem. Databases are especially good for saving large amounts of data in a single location. You can run sophisticated queries on data. For example, you might want to analyze data acquired between certain dates or compare data from one date period to another to observe trends. In my situation, all test data is written to a large SQL Server database. Users can log into our intranet and obtain real-time yield data, nothing can ship without a pass record, customer service can look up a unit to determine when it was tested, what if any tests failed, what repair actions were made against it, etc. Powerful tool but it requires a pretty large investment in hardware and software support.

I hope I haven't just confused you with my explanations. In your specific situation, I think you'd be better off with doing some sort of text file saving. In the short term, anyway, you'll probably find that it's easier to get that up and running sooner.

[DFGray]

Be careful with text data storage. I agree that it can be fast to get up and running. It is also compatible with anything, cross-platform or not. However...

If you are storing floating point numbers, be sure to store enough digits of precision. This can really ruin later post processing. This is a real problem with the LabVIEW native XML, since you have no control of this.

If you want random access to the data in the file, it will be very slow, since you need to parse everything before an item to get to the item you need. If analysis speed is an issue, don't use text.

The best solution I have used (many, many times) is HDF5, a self-describing, hierarchical, binary file format. You don't have to remember the format - it is automatically included in the file. You have hierarchy so you can put data of arbitrary complexity in the file. It is binary, so speed is usually hardware limited. HDF5 is a shared library (DLL on Windows) and is available on any platform LabVIEW is available, so you can analyze your data from other programming environments. Best of all, HDF5 is free, since it was developed by the NCSA for supercomputer users (your tax dollars at work). Now for the down side. HDF5 is very low level, so it takes some effort to use it. HDF5 is not multithread safe on Windows, so you need to be very careful in LabVIEW. If this does not scare you, you can check out the Windows LabVIEW port (called sfpfile). The HDF5 main website is here.