Allocating enough memory to open a large data file

[altenbach]

A ~300MB file of DBL gives about 40M data points for the two columns. This means you want to do a FFT of about 20M points. Giving a 40MB complex output.

Initializing a 20MB DBL array and trying an FFT gives me an out of memory error on my 1MB system that has been running for a while. I'll try a more massive computer later.

Is this what you have?

I am sure that FFTs like this can be done in pieces if you chop it up right, you'll only lose the very low frequencies that you can reconstruct in a different way. What frequencies are you most interested in (low, itermediate, high)?

Can you tell us a bit more about your experiment?

[Drewski]

You've mentioned several times that your system has 3Gigs of Ram.  Is it a dual CPU system?  Is the ram divided up amongst the processors?  How do you know it has 3Gigs of ram? When you look at task manager, how much virtual memory is being used? how much memory is free.

Just because you think you have 3 Gigs of Ram doesn't mean it's all empty.

Why do you refuse to read in a portion of the file?   If you could read in a portion of the file, then a little more, then a little more until you run out of memory, that could give an indication of how much ram is available.  Can you read in 100 megs?  50 megs? 1 megs?  When do you start to get the error?

[altenbach]

Reading a 300MB DBL file should not be a problem unless the memory is highly fragmented. Taking an FFT will easily need multiples of that, so reading the file is only the minor part of the problem. Constantly trying to read variable sized chunks, will only fragment the memory more.

As has been mentioned, arrays need to be contiguous in memory.

[BrockB]

I've actually switched over to singles per your suggestion, so it's actually about 40M data points per column.

The data is an interferogram from a Fourier Transform Spectrometer. We're trying to replace the electronics on an old DA8, and the first step in that process is to piggyback our own digitizer onto the existing system and take data simultaneously to see if we get the same spectrum. The existing system uses the zero-crossing points of the metrology laser interferogram to trigger data acquisition in the detector. We'd like to simply digitize everything in sight (ie both the metrology laser and detector signals) and process the data later, as my supervisor believes this may give us more accurate results. The metrology laser interferogram has a frequency of about 16kHz when the FTS is run at 1cm/s, and while normally it would only take a sampling rate of twice that to be able to characterize it, it isn't that simple: The carriage's speed isn't perfectly constant, so the frequency of the interferogram changes, meaning that we have to sample at a much higher frequency if we want to ensure that we consistently get measurements sufficiently close to the zero-crossing. Right now, we only need to do an FFT using points at the zero-crossings, however later on I would like to be able to interpolate the data to get an evenly spaced set using all of the data points in order to get a more accurate spectrum. Granted, it won't improve the accuracy by much, but the data is there so why not use it?

I am starting to get frustrated though because this is a problem which I shouldn't have to work around by opening the data file in parts, etc. We bought a new computer specifically for the purpose of doing this experiment. It has 3GB of memory, only 23% of which is supposedly being used right now. We *should* be able to easily open and manipulate 300MB or even 600MB data files.

To answer your other question, the processor is a Q6600. I don't see how it being a quad core should affect memory allocation, however

Message Edited by BrockB on 11-12-2008 01:12 PM

[BrockB]

If fragmentation of my system's memory is a problem, then how do I fix it? I've already tried rebooting the system and running the program as soon as the system came back up, but it didn't work. I suppose I could try a hard boot if that is more likely to completely clear the memory.

[altenbach]

---

BrockB wrote:

I've actually switched over to singles per your suggestion, so it's actually about 40M data points per column.

---

I would not do that since the FFT only accepts DBL. Reading it as single would cause a coercion and thus another copy in memory.

I don't have time to look into the rest at the moment. Maybe tonight.....

[altenbach]

---

BrockB wrote:

It has 3GB of memory, only 23% of which is supposedly being used right now.

---

DId you tweak 32bit windows so it can actually use all 3MB? See this note.

[TCPlomp]

Why do you absolutely need to fft the whole file in once?

Again what frequencies are the ones you are interested in and what is the resolution you need?

What code do you use for the FFT?

Ton

[BrockB]

Funny you should mention that virtual memory page. I actually just found that in the help files, and after rebooting, it seems to have helped - I've been able to open one of the 300MB files and analyze the data. Since the FFT uses doubles though, I'll have to switch back and see if it still works with the 600MB files.

With respect to the FFT, I haven't programmed that part yet, but I had planned on using the FFT vi included with labview. We'll be mostly looking at the infrared, and the highest frequencies we're interested in will be less than half that of the metrology laser. As for why I need to use all of the data for the FFT, I really don't see how you could do a fourier transform in parts.

[BrockB]

It did work with doubles as well, but I got an error when trying to write the modified data to a new file, so I ended up having to write it one point at a time, as before. I also got a new error at the beginning, telling me it had hit an unexpected EOF. I did change the data type wired to the read binary to double, and changed the number of bytes per data point to 8 so it shouldn't have. Below is what the read-in portion of the vi looks like. When I hit continue, the rest of the vi seemed to execute without problem though, once I set it to save the data one point at a time instead of all at once.