When you say input probe, do you mean your input channel? If so, it is not possible to isolate your PXI-5112 scope from the chassis ground. Please see the link below for more information on isolation techniques and other isolated National Instruments products.
High-Voltage Measurements and Isolation
http://zone.ni.com/devzone/cda/tut/p/id/3410
If you are trying to do AC or DC coupling with your PXI-5112 then this can be selected through the software. You can see how this is implemented in the niScope EX Configured Acquisition.vi example. Please see the NI High-Speed Digitizers help for more information.
NI 5112 AC and DC Coupling
http://zone.ni.com/reference/en-XX/help/370592E-01/digitizers/5112_coupling/
What kind of problems are you having when you say you are not getting consistent measurements? Can you give more details about how exactly you have connected the signal and what exactly you are seeing?
Regards,
Chris Delvizis
National Instruments