LabVIEW
Allocating enough memory to open a large data file
There is no need to get the file size, just set the count to -1 and it will read the entire file.
One problem is still that you read twice as much as you actually need. You could allocate the final size of the desired column into a shift register, read&decimate in parts, and fill the column data "in place".
If you're feeling brave...
...and you're still up against memory limitations, you might need to consider using a different FFT implementation. The built-in LabVIEW function is very convenient (and suitable for 99% of cases), but it's had to be designed to be flexible enough to cope with anything that a user might throw at it. For a single use-case you should be able to optimize this significantly:
- FFT algorithms can inherently be done 'in-place' (without taking a single memory copy). The NI version takes at least 1 copy so it can cope with changing array lengths
- You can obviously implement it with singles, not doubles
This should give you (at least) a x4 reduction in the memory required. There are lots of algorithms out there on the web that you can just nick - my favourite would be from "Numerical recipes in C" (because it gives a good explanation of how the algorithm is derived). You could try implementing this as a DLL call, or it'd be really interesting to see a pure-G solution!! Unfortunately this is only a suggestion, as I don't have time to try it out myself. 
Also, are you sending 2^n samples to the FFT (even to the built-in version)? If not, it can't run as efficiently - you could end up with it doing a "slow" FT.
Ian
Ian C wrote:Also, are you sending 2^n samples to the FFT (even to the built-in version)? If not, it can't run as efficiently - you could end up with it doing a "slow" FT.
The NI version have been completely rewritten quite a few versions ago and are now very efficient, even for non-integer powers of two sizes. The dependence on input size is no longer as dramatic as it once was. Still, integer powers of two sizes are typically the top scorers. Long ago, I did a lot of benchmarks with variable input sizes.
The NI algorithms compare quite well to e.g. FFTW but of course we only have DBL. A set of DLLs for FFTW are freely available for download.
Quote: "We have created precompiled DLL files for FFTW 3.1.3 in single/double/long-double precision, along with the associated test programs"
I don't know how easy it would be to call those DLLs in LabVIEW, but I am sure it's doable. Somebody could do a nice toolkit of SGL LabVIEW VIs with them...hint... hint..! 😉
Another option might be to adapt a fixed-point implementation as used inside FPGA I don't know how that would compare.
Very long ago, we had an FFT challenge (discussed here) to try to bet the NI implementation, but I don't think it produced any results.
