Processing very long recordings

[golubovski]

Hello,

I will have very long acquired arrays of channel data (tens, possibly hundred of megabytes sized files) that will need processing like FFT for example. Are there known standard procedures or algorithms for sequenced FFT - such long arrays are impossible to handle at once in the memory...?

Thanks in advance,

Roman

[Charlie_P]

Good Afternoon Roman,

As you correctly understand, you cannot load part of an array into memory; it is all or none.  Is there a particular reason that you are storing these values in an array instead of directly to a file?

It seems that you will be best off writing your data directly to a binary file without the arrays.  This will give you the speed associated with a binary file, as well as the ability to read part of it at a time, keeping your memory usage to a manageable amount.  There are many examples of File I/O in the NI Example Finder.  (Browse according to Task, Fundamentals>>File Input and Output.)

[golubovski]

Charlie,

I apologize for the misunderstanding - I do want to access very long files sequentially!!! My question is - how to perform common processing on whole data when having chunks of it in the memory at a time? How to do FFT - is there a proper procedure, do I need to do correlation of any kind, averaging, segmentation...?

Thanks in advance,

[Charlie_P]

Good Evening Roman,

I can't think of any particular method that we generally recommend.  The algorithm that you follow will need to be based on your processing needs and computer resources.

[CoastalMaineBird]

Do you really need to do a single FFT on a file that's hundreds of megabytes long?

That's a very odd requirement.

A Gabor spectrogram will give you amplitude vs. frequency vs. time.  It basically takes a chunk, performs an FFT,  slides the "window", takes another chunk (which may overlap the previous chunk), performs an FFT, etc., etc.  That tells you not only the amplitude at a given frequency, but WHEN in the file an even occurred.

You can do a DFT (Discrete Fourier Transform) on such a file, if you really really really need to.

For each sample:

   S = sin (2 * pi * F * t)

   C = cos (2 * pi * F * t) 

   R = S *  V[i] + C * V[i] i  

   RTotal = RTotal + R

end for 

   where F is the frequency you're looking for.

   t = the time of the sample (= sample # * sample period )

   V[i] is the input value for sample i.

   R is a complex number 

and at the end of the loop, Divide RTotal by N, and you have the complex coefficient ( amplitude and phase) of the signal at frequency F.

That lends itself to processing a sample at a time, but I still question the need for it. 

Message Edited by CoastalMaineBird on 05-09-2009 11:36 AM