Quite easy.
Every time you do an average, grab the millisecond timer value and store it in a shift register. Then subract the value in the shift register from the current value, and you will always have the elapsed time in milliseconds. Convert to seconds or minutes as appropriate for your situation.