Sorry for the confusion. Since the object I'm trying to measure is round, I'm using the IMAQ Spoke to locate the edges. In this case, the real profile is what I'm interested in, so I can't just fit the data to a circle - I need to know where each edge point. I'm trying to make an edge profile or contour of the object. I had considered using an edge detection filter (Canny, Sobel, etc) but I don't believe I can get the accuracy I need with such an approach.
The stair-step effect is happening in the profile of the edge. I know I have a smooth, very circular dome that I'm imaging. Therefore, the edges should form a nice, smooth circle. However, I'm getting a pixelated or stair-step appearance when I plot all of the found edges in their x,y pixel coordinates. I had assumed that if I used sub-pixel resolution that these pixel-based artifacts would be reduced or eliminated. When I plot out the dome and compare the found edge locations to the best-fit circle, I get what looks like very high frequency oscillation of about 1-2 pixels around the average radius. (I convert the data to polar coordinates with the "best-fit" circle cneter point as the center of the data. The radius values can then be compared to the average radius very easily)
One approach I'm considering using is just a smoothing filter on the profile. However, I'd rather solve the problem in the edge finder since that is the real source of the problem.