Hello Ken,
You have two scenarios to start from - either knowing ahead of time what the expected noise level is based on your settings, or not knowing. Also, whether the displayed noise level is the system noise level (i.e. the signal noise is lower than the system noise) or the signal noise level. Of course, is the noise level is the system noise level, you essentially know ahead of time what the noise level will be based on the 5660 specs.
If you are looking at system noise as the noise floor, you can look at the 5660 specs and calculate the noise floor based on the noise density value, 5660 RF attenuation, and RBW.
Noise Floor (dBm) = Noise Density (dBm/Hz) + 10*LOG10(Actual RBW) + RF Attenuation (dB)
If the noise level varies and your application has to adapt, then you can use the power spectrum. If you want to average a section of the spectrum (that contains no noise), you need to do so in linear units, not dB or dBm. A quicker way to do this would be to do a simple max value test for a portion of the spectrum noise that doesn't contain signal and then set your threshold 3 dB above this.
Regards,
Andy Hinde
National Instruments