Hello Grace,
"(1) Is the Out signal TTL/CMOS?"
Yes, the outputs are TTL Signals.
"(2) What is Output impedance?"
The Output Impedance is 75 Ω (56 Ω from an onboard resistor and 19 Ω from the TIO ASIC).
"(3) When I monitor a signal with oscilloscope, the output voltage is 5V with 1 MOhm DC coupling, but less than 2 V with 50 Ohm DC coupling. My thought is that the output voltage should be ~5 V in 50 Ohm coupling."
Because the output impedance of the 6602 is 75 Ω, a 50 Ω load will sink more current than the 6602 can provide. This causes the output voltage to drop to below 2 V. To prevent this, avoid using a load with such a small impedance. If you must use the 50 Ω coupling, you should put a large resistor (in the kΩ range at least) in series with the load.
I hope this helps.
Regards,
Sean C.