Hello;
Differential Mode is always the best mode to use. Make sure you inserted a BIAS resistor in between the ch- and AIGND to create a path to ground for the signal. That might be the reason for the noise you are seeing there.
Regarding your question, RSE and NRSE are alike. The main difference is, if your signal source is referenced to GND, you can create a ground loop and damage the DAQ device, if you use RSE.
NRSE, pretty much will adopt a reference, that is not ground, to reference all the signals. So if your signal sources are offseted from each other, you might see an offset on your readings.
Again, the best method is Differential Mode. I suggest you to double check the placement of a BIAS resistor on it. 100K ohms is a safe choice.
Hope this hel
ps.
Filipe A.
Applications Engineer
National Instruments