Hi, I am using DIAdem for the FFT on the vibration data. Total number of data rows are around 70 million and 4 columns, sampling rate is 96000 samples per second. Size of my RAM is 3.25GB.

[Etn]

Hi,

I am using DIAdem for the FFT on the vibration data. Total number of data rows are around 70 million and 4 columns, sampling rate is 96000 samples per second. Size of my RAM is 3.25GB.

When I am doing the FFT for any 1 of the channel its showing that there is not enough memory for the calculation. But when I am doing any other mathematical calculation like finding the Standard deviation or mean etc it’s able to compute  for the same data.

I just wanted to know what will be the probable problem for doing the FFT and how I can resolve this problem.

Thanks,

Deepak

[dsb@NI]

Unlikely that you need .001 Hz resolution, so could you use averaging to break up the FFT over several blocks and still get an overall result? You can compute an averaged FFT on a channel of 70 million samples (or more) in DIAdem. In the FFT Settings, set the Time Intervals... options by setting the block size to achieve the required frequency resolution (at sample rate of 96k, block size of 65536 yields frequency resolution of ~1.5 Hz).

My machine (RAM = 4 GB) was also not able to compute the FFT on 70M samples in one shot. The FFT is more processing intensive than the mean or standard deviation. If you really need a 35M-bin spectrum, you will have to get creative.

[Ulrich_Bierwisch · Accepted solution]

Hi,

Calculations like Standard Deviation or Mean are going throu the channel from start to end. They don't need much additional memory to perform the calculation.

The FFT is different. The algorithm needs several additional channels to store the values (as fas as I know 5). Each channel has to be allocated in one peace. On a 32-bit Windows, a single application can address a maximum of 2 GB of RAM. This is cut into smaler parts and depending on the situation and what you have done already you can get more or less memory in smaler or larger peaces. On a typical modern PC like yours, the maximum size for the FFT might be between 10 and 30 million values.The situation can be improved on a 64-bit Windows. DIAdem is still a 32-bit application, but Windows can address more RAM. 70 million is still out of reach.

There are several solutions, depending on your needs.

If you need the high frequency amplitudes (up to 48 kHz) over the whole time of the measurement, you can perform the FFT in several steps with averaging. 10 FFT's with 7 million values each can be calculated very fast and the average is still an FFT with impressive resolution.

If you need one FFT over the whole time because you need the low frequencies in a very high resolution you can reduce the samples per second (after performing a low pas filter).A reduction by the factor of 10 will result in an FFT up to 4,800 Hz with full resolution.

Only if you need high frequencies in high resolution, for example the difference between 40,000 Hz and 40,000.002 Hz, you need one big FFT over all values. DIAdem might have a solution for this too.

DIAdem has an old compatible FFT-algorithm which was important when memory was much more expensive. This algorithm can store the FFT-values on file and is not limited to RAM-allocation sizes. But the algorithm is limited to numbers of vales which are a power of two. In you case the FFT can calculate with 67,108,864 values. If the channel is larger the rest of the values is ignored. If the channel is smaller, the next size is 33,554,432.

You can switch to this algorithm in the Preferences menu for compatibility. The calculation might take a very long time.

But it's going to be even better:

I have tested the FFT with the latest beta of DIAdem 11.1. DIAdem seems to be able to run much bigger FFTs because of an improved algorithm which can calculate the FFT in chunks. Unfortunately I'm not able to confirm 70 millions because the calculation takes to much time to finish right now. 

Ulrich Bierwisch

[LabVIEW Freak]

Hello Doug,

Please correct me if I am wrong, I think you mean we can simply mention the block size in the encircled area? Please refer to my attachment.

Regards,

Guru

[Etn]

Hello Doug,

As you mention that to change the block size for doing FFT, but I did not got any option of Block size in Time intervals. Could you please let me know what do you mean by Block size? For the same issue I have talked to NI people in india but they are alos not able to provide me the right information. Could you please explain what do you mean by that?

1 more thing I did not got the meaning of averaged FFT, as much I know about FFT it has to be done in single run becuase there is chance of geeting wrong answer for this.

Deepak Singh

[Etn]

Hi Ulrich, I have tried this on 64-bit Windows but still I am not able to do the FFT on single run.  As you mention that perform the FFT in several steps with averaging. I didn't understand the meaning of averaging? As much I know about FFT on any time signal, doing the FFT not on whole data as whole in single run will not give the right answer. Please provide any reference for the same. I have already finished the test with 96000 sampling rate and having 70 million data point. Now doing the reduction of sampling rate by 10, mean sampling rate by 9600 will not affect the FFT? Please provide any reference so that I can go ahead with this. I have tried the third option but still I am not getting the solution. Thanks,Deepak Singh

[Ulrich_Bierwisch]

Hi,

Averaging means, that you are not calculation one FFT over all Values but for example 1000 FFT's over smaler peaces of the signal.I have attached screenshots from the dialog of the latest DIAdem-Version how this can be done.

Instead of one result with 35 million values from 0 to 48 kHz, which is difficult to handle, you get 1000 results with 35,000 values from 0 to 48 kHz. Usually, a signal is changing over time so the results are not identical.If you choose "Average amplitudes: Arithmetic" like I did, you will get one averaged FFT over the whole signal. I think this is what you want. The calculation is fast enough so you can change the number of intervals from 1000 to something else to see the difference and decide what is best for your needs.

It is possible to choose "Average amplitudes: No averaging". In this case you get 1000 FFT's and you can see how the FFT is changing during the time of the test.

You second question:

If you reduce the sampling rate the result in the lower part of the FFT is not affected and looks like the result of the FFT over all. This is the way to go if you realy need the high resolution (which seems to be not the case here).

[dsb@NI]

Yes, Interval length = block size

[dsb@NI]

Any response to Ulrich's comments? What range of frequencies are you interested in? What frequency resolution is required? Are you interested in a single FFT with high resolution, the averaged spectrum, or how the frequency content changes over time? Your concern that the FFT would yield inaccurate results really depends on the nature of the signal. Some more details on what you are trying to learn from the data may help us offer better suggestions.

[Etn]

Hi Doug,

I am on leave for few days. Next week I am going to work on the suggestion given by Ulrich.

However the Input which we have given in our component is 40G with frequency varying from 10 Hz to 4000 Hz with sweep rate. of 1 octave per min. Actually our component is electromagnet sensor and its response is affected by the vibration amplitude and the phase. We are trying to find at what are the frequency our component is giving the most undesirable response as per our pre assumed output.

Deepak Singh