Logging Raw Data

[Alv]

I'm trying acquire raw acoustic data for 10 channels, store it to file, read it and then do my 1/3 octave analysis. This required since my buffer isn't large enough to handle the necessary post-processing (Calcs) for the numer of channels needed at one time. So, I'm not sure how this is done or what format the files are stored (logged) or how read. Does anyone know how to go about this effort?

[DFGray]

You have several options, depending upon your performance, ease-of-use, and maintainability requirements.

If you have LabVIEW 7.0 or greater, the easiest thing to do is use the LVM Express VI to stream data to disk. You can also read it back using the LVM Read VI. LVM is a text spreadsheet format, so is very slow and very compatible with just about anything for reading it. You won't lose your data by forgetting the format. However, from your brief description, I would not recommend this due to the low performance and difficulty to read single columns back in (LVM caches the whole data file in memory when reading for performance reasons - it assumes low point counts).

Option two is to stream data to disk as an interleaved string of binary values. This is your highest performance write option, but readback of a single column is somewhat difficult since the data is interleaved. A variant on this is to stream to 10 separate files at once. It will be slower, but read back is much simpler.

Option three, if you have access to it, is to use NI-HWS. This is probably your best option, since it allows you to store 10 independent waveforms in the same file at high speed and retrieve them by name. Unfortunately, it is only distributed with the computer-based instruments (scopes, arbs, and high-speed digital), so you may not have it.

Option four, if you are into pain, is to use the precurser to NI-HWS - sfpFile. You can get this from ni.com by searching for it. It offers more functionality than NI-HWS, but is much harder to use. Both it and NI-HWS are based on HDF5, a self-describing, cross-platform, binary file format developed by the National Center for Supercomputing Applications specifically for fast, hierarchical data storage.

What you pick will depend on what you really need. Answer these questions and you will be on your way. What speed do you need to save data? Are you constantly streaming or taking the data, then saving it? How do you need to access it when you are done? Will you need to access it several months from now when you have forgotten how you saved it?

If you need more help, post more info and we can move you in the right direction.

[Alv]

Thanks for responding,
I'm not up-to-speed on any of those methods, but what speed do I need to save the data? I assumng your talking about sample rate, 51.2KHz (10-20kHz 1/3 Octave) or as low as 44kHz. 2nd question, I would take enough data to get a good average (say 3-5 minutes worth) and then save it to file (excel). 3rd Question, I retrieve the data in excel (spreadsheet file...*.txt). 4th question, I will all ways need to retrieve the data for future reference (or maybe not, because I'm recording everything on a Digital tape recorder). I'll look into the last question.

Scott

[DFGray]

From your answers, it appears you are already happy with a text based solution. You have two options:

Use LVM files
If you have LabVIEW 7.0 or better, there is an Express VI set which will make your life very easy. They are Write LVM and Read LVM respectively. They can be found in the Output and Input Express palettes. Just fill out the dialogs and wire your 2D data directly in.

Use Write to Spreadsheet File.vi
If you have an older version of LabVIEW, you can use the Write to Spreadsheet File.vi found in the File I/O palette. This one is almost as easy to use, just wire your 2D data into it. You can write the data in chunks. You may have to transpose the data before writing it if the columns and rows are switched in your data file (us
e the Transpose 2D Array.vi from the Array palette). You will probably also want to add some header information so you don't have to remember what is in the file. Check out the examples that come with LabVIEW for how to do this.

Good luck. Let me know if you have any problems.

[Alv]

Using the LV measurement file vis will require me to give these files names. All I want to do is store the 10 channels of data, puase or stop acquiring the data, then read the data back into the octave analysis vi/power spectrum vi (which are in a case structure) to do my analysis. Can all this be done while my code is running? If you like I can send an example of my code.

R/Scott

[DFGray]

You don't really need a file, then. Just wire the data from your acquisition to the analysis VIs. If you want to acquire while you analyse, try using a queue to pass the data (queue VIs can be found in the synchronization VIs).

If you post your code, I can probably make more constructive comments. ZIP it with directories intact if you have any sort of directory structure.

[Alv]

I've tried writing to LVM file. I couldn't get it to work properly. Not sure if the DAqmx, scale Volt vi, and write LVMvi should be outside the loop. Need help. Is the writing to LVM.vi taking a snap-shot of data (in other words, is this like a digital recording of data?). If so, How can I take several minutes of data? I really don't have a preference if the file is txt or binary. Once I analyze the data, I write the data to file (txt) anyway.
I've heard of queue's but don't know how to use them. I've included an example of my code. You see some vis that are there just for writing to file but may not show up. Doesn't matter though. Let me know if you have problems.
R/Scott

[DFGray]

My apologies for being slow getting back to you. I got some high priority interrupts and was unable to look at your code in a timely fashion.

At first glance, I have a few comments:

1. At the moment, you are querying for file name and writing a new data file every iteration through your loop. If you replace your query and write to spreadsheet file with the Write LVM express block, and set it up correctly, this problem will go away. Set up the Write LVM block to Save to one file->Ask user to choose file->Ask only once and If a file already exists->Append to file.


2. A second method is to initiate the file on interation 0 of your loop (is loop counter 0?) and append to it on every loop afterwards.
   It doesn't matter if you write to the file on iteration 0, but you do need to open it. If you use this method, I would recommend NI-HWS from the DriverCD, as it will have much better performance than LVM files (NI-HWS files are binary). You can also set up the file open/close to happen on user demand, but that takes a bit more code. See the next couple of comments.


3. You need to use the event structure to make your user interface work better. Check out Managing Large Data Sets in LabVIEW, the GLV_TypicalMemoryStoreAndBrowse.vi for an example of using the event structure while running an acquistion loop. You would place another case in the bottom loop containing your acquisition loop.


4. You need to be more careful about data copies. Read the rest of the tutorial I just referenced. It is fairly advanced stuff, but you can do it. Walk through the examples as yo
   u read it.

Your main problem is you are creating/overwriting a new file at every loop iteration. If you fix that, you should be able to go forward.

[Alv]

Hey thanks for looking over my code. I'm new to LV so, please excuse my first go at this.
With regards to statement 1, I have my writing to files in a case structure (true). Should this be false? I thought none of that gets excuted until I press the button to write the file. In addition, there are some corrections to my data when I write to file also (sequence structure, file is written, file is retrieved and added to and file is renamed). I had LV asking me what file to write etc, but was a cumbersome process. I thought by putting these writing files, retrieving files into a sequence structure would make it easier on the user. Is this causing my buffer problems? I'll read over the link and I'll be back wit
h you. Thanks. I think I need a LV Buddy.