Hi Terrill,
Your approach to this measurement is correct. Doing a coarse scan to find the peak in the spectrum will get you a coarse measurement of the frequency. At that point you would use the coarse measurement as your center frequency and zoom in by reducing your span and resolution bandwidth (RBW). The resolution bandwidth input is what will affect the accuracy, but to achieve a satisfactory measurement time with lower RBWs, a lower span is necessary. I would suggest a span <= 1.25 MHz, and the smaller the better. A span <= 1.25 MHz will ensure the PXI-5660's digital downconverter (DDC) present on the digitizer module will be enabled, thus accelerating the measurement greatly. Lower spans achieve even more DDC 'acceleration' - see the PXI-5660 documentation for more info.
When the DDC is enabled, you can use lower RBWs. The RBW setting determines the acquisition time, or amount of data sampled. This is known as 'gate time' in frequency counter terminology, so lowering the RBW in effect increases the gate time, which increases the accuracy of the measurement up to a point. I am attaching a VI which uses PXI-5660 specifications to calculate the expected frequency measurement accuracy you can expect based on the signal frequency and gate time of the measurement.
So in summary:
Span should be <= 1.25 MHz, and the lower the better because the lower the span, the more DDC help you get.
DDC help is important because greater measurement accuracy requires longer gate times, which is in effect the same as lower RBWs. Lower RBWs require more processing which the DDC helps.
The attached VI tells you what frequency measurement accuracy you can expect with the 5660. Best case is right after a fresh calibration. Worst case is after worst case aging specs of the onboard timebase after a year's use with no re-cal.