Do you know a more efficient way to store and retrieve a waveform?

[3d0g]

I looked a bit earlier, here, and I didn't read of a better way, so I figured I'd just post the question and see where it goes.

Presently I'm successfully storing and retrieving waveforms that look like this:

------ begin snip paste ------

620.555200E-3, -726.576206E-3
620.556000E-3, -726.572752E-3
620.556800E-3, -726.569299E-3
620.557600E-3, -726.565845E-3
620.558400E-3, -726.562391E-3
620.559200E-3, -726.558937E-3
620.560000E-3, -726.555484E-3
620.560800E-3, -726.552030E-3
620.561600E-3, -726.548576E-3
620.562400E-3, -726.545122E-3
620.563200E-3, -726.541668E-3
620.564000E-3, -726.538214E-3
620.564800E-3, -726.534761E-3
620.565600E-3, -726.531307E-3
620.566400E-3, -726.527853E-3
620.567200E-3, -726.524399E-3
620.568000E-3, -726.520945E-3
620.568800E-3, -726.517491E-3
620.569600E-3, -726.514037E-3
620.570400E-3, -726.510583E-3
620.571200E-3, -726.507129E-3
620.572000E-3, -726.503675E-3
620.572800E-3, -726.500221E-3
620.573600E-3, -726.496767E-3
620.574400E-3, -726.493313E-3

------ end snip paste ------

Above shows just an (relatively) infinitesimal snip of the waveform (txt) file.  (It's near 72MB of such.)  It takes about 8 seconds to read in the data from the file.

Later I intend to be storing the FFT data, the real, and imaginary parts.  (The capture parameters are fixed; otherwise, I'd need to store that as well.)  In the meantime I'm trying to think of a better, more efficient method than this.  This one works fine, but it's slow.  The above is easily dropped into Excel.  The latter (the FFT) will be droppable, too.  However, I could see running some sort of a compressed file through a function that would produce something Excellable; that'd be reasonable.  But what would that compressed form be?

I ask you fellow LabVIEWers.

Thank you

[3d0g]

If it wasn't clear, the above snip is [t, f(t)] data of the waveform.

The FFT data will look the same as the snip, but it'll be the [Re(f), Imag(f)] data.

I'm presently using the filenames themselves to state the capture parameters.  I can parse the filename, either visually or LabVIEW, to pull out the information.

[Kevin_Price]

This might become pretty straightforward (and likely considerably faster) with a TDMS file.   NI offers a plugin for importing TDMS files into Excel.

Where does the data come from?   It looks like constant-rate sampled data at 1.25 kHz (800 microsec time between samples).  So most of the time column is pretty redundant.  TDMS could store that as a waveform, saving the full array of f(t) values but only saving the delta t one time.

Similarly, the FFT data could be put into waveform format using a complex DBL datatype to store the Re and Im parts in a single value.  The delta freq would again only be saved once.

If you're using a DAQmx device, there's an option for DAQmx to automatically log your data to a TDMS file in the background as your task runs.  You could read it back in when it's done, compute your FFT, and then add the FFT results to the same file.  

-Kevin P

[santo_13]

ASCII is not best at storing data efficiently, for example, 14 ASCII characters to represent a floating point number which takes up 14 bytes, whereas if stored in binary format will take up only 4 bytes ~70% smaller, so your 72MB would have been actually 20MB and would read faster.

[3d0g]

It's 1.25MHz sample rate over a sample time of 2 seconds of a 1Hz sine wave, but only a snip of it.

I understand the time captures are redundant; however, I see this as a model of what the complex numbers would look like.

The TDMS idea has my interest.

The binary option has even more of my interest. 

I would need a conversion function for either.

I went with ASCII to prove my concept (which I think I've done to some extent ...still testing.)  Now I want to improve my speed.  The FFT time isn't an issue.  It's this data loading time that I want to fix.

Also, I'm seeing evidence that LabVIEW is being slowed for some reason.  That's weird, because I've set that optimization environment option from 1 to now 0.  Why is LabVIEW slowing?  Have I really taxed LabVIEW this much?  I didn't think so, but maybe so.  I slide the scroll bar and it'll take many seconds to move, at times.  (I close LabVIEW and then restart and return to my project to reset the issue.)  I'm still trying to determine the conditions, determine if it's my error creeping in somewhere.  I am holding onto test arrays; LabVIEW keeps that data such that it can be examined outside runtime (during development.)  Maybe that's the culprit somehow.  (I hope I'm using these terms correctly by the way.)

[santo_13]

---

I went with ASCII to prove my concept (which I think I've done to some extent ...still testing.)  Now I want to improve my speed.  The FFT time isn't an issue.  It's this data loading time that I want to fix.

Also, I'm seeing evidence that LabVIEW is being slowed for some reason.  That's weird, because I've set that optimization environment option from 1 to now 0.  Why is LabVIEW slowing?  Have I really taxed LabVIEW this much?  I didn't think so, but maybe so.  I slide the scroll bar and it'll take many seconds to move, at times.  (I close LabVIEW and then restart and return to my project to reset the issue.)  I'm still trying to determine the conditions, determine if it's my error creeping in somewhere.  I am holding onto test arrays; LabVIEW keeps that data such that it can be examined outside runtime (during development.)  Maybe that's the culprit somehow.  (I hope I'm using these terms correctly by the way.)

---

I would not say LabVIEW is the fastest out there but it is neither slow, it is still very competitive in terms of performance. With that said, all that you are describing seems to very much dependent on your implementation and has nothing to do with LabVIEW by itself. Please provide example to back your comment and the fellow forum members would be glad to explain the performance and even methods to make it better.

This is a nice article - https://www.ni.com/en-us/innovations/white-papers/09/comparing-common-file-i-o-and-data-storage-appr...

[altenbach]

First of all, this is not a waveform, but xy data where x seems to be time.

1. The time is fully defined by two scalars (x0, dx), so storing the first column is redundant. All you need to save is x0, dx, and [y]. (and you can even omit data that are defaults, e.g. is the data always starts at t=0. You can also encode dx in the filename). This already gives you a 50% saving.
2. Formatting floating point numbers  into decimal strings and scanning a decimal string back into floating point datatype is expensive AND lossy AND wasteful. If speed matters, you need to go binary.
3. What does the data represent and where does it come from? Maybe you could save the unscaled binary data from the digitizer. Might fit into U16, for another 75% saving over DBL. Even SGL might be sufficient for the scaled data.
4. Compared to anything else, disk IO is relatively slow, especially if you have a conventional HD.

You talk about FFT later, and that would work on SGL too. for FFT all you need is [y] and (dx to calculate df). the timepoints are not needed. Why would you need to store the FFT output RE/IM if it can be recalculated at any time from the data?

Can you explain in more detail what you are actually trying to do and where the speed constraints/requirements are. It seems you should speed up the saving and the retrieving! Are all these files just for your use or are there requirements to be able to read them later using third party software?

[3d0g]

I want to capture the full audio range with a df of 0.5Hz and some appropriate window over the sample.  (I view a sample as a sum of sinusoidal f(t)'s, and I've been working with 1Hz, 0.5Hz above my lowest. ...easier read on my test plots.)  ...I had to put a limit somewhere, so I chose df=0.5Hz and the sample rate is good for my PicoScope.

I like the binary storage idea and will be pursuing it, first.  That may be all that's needed.  I envision a simple substitution in the VI, based on the help info I've read.

For the FFT I need the phase information as well.  It seemed reasonable (to me) to store the FFT as R +/-jX.  It'd appear, based on the above, that can be done via binary storage of LabVIEW's complex double.  I intend to find out if it'll work, shortly.

Many thanks for posting an article for me to read!

[3d0g]

Oops! Make that A +/-jB.

[3d0g]

Binary waveform files 20MB. (19 and change)

Speed is excellent, at maybe 1/2 second to load.  Saving the file has improved tremendously as well: a couple seconds, max.

Thank you, all.