DFGray suggests an excellent approach to creating an optimized solution to this problem. However, if you just need to do this once (or occasionally), and speed isn't that much of an issue for you, you can write a very simple program that sequentially reads data, processes it, and stores results in a shift register, rinse and repeat. It will take a few hours to process the data. For 50GB of data, the difference between the two techniques will probably be on the order of 10's of minutes.
On the other hand, if this is part of a continuous process where speed and efficiency are really important, then I would spend some serious time parsing every part of DFGray's post.
Chris
