Well... This is not as simple as it sounds.
In order for you to get a reasonable approximation of the system noise, your waveforms must be of the same amplitude, frequency and phase and your sample arrays must be of the same length.Since you are sending the source signal through the system and then trying to compare it, you are going to have to deal with system delay (which produces a phase shift that is going to be frequency dependent) and any possible distortion of the waveform through your system.
Doing a simple subtraction of the input waveform from the output is only going to be a gross approximation, not a precise measurement. The problem is that you are not going to be able to correct for any harmonic distortion that may be present on the output signal that may not be there on the input.
What you really want to do is measure the broadband output RMS voltage (or power) of the system output signal with no input signal applied and the input terminated in the characteristic impedance of the signal source. This will give you the noise floor for the system. You can then measure the RMS voltage (or power) of the output signal with the input signal applied and using the appropriate formula below, get the signal-to-noise ratio (SNR) of the system.
SNR = 10*Log(Signal Power / Noise Power)
SNR = 20*Log(Signal Voltage, rms / Noise Voltage, rms)
Now be aware that this is still only an approximation because your measurement instrument and any signal-conditioning you do are also going to add some noise in the form of quantization noise in the ADC, thermal noise in any active and passive signal processing components and any crosstalk or pickup of external EMI sources. If the noise contribution of your measurement system is at least 10 dB below the noise floor of your system under test, you will get a reasonable approximation, if not, you will have to correct for the measurement system noise contribution.
You must also be cognizant of the fact that your ADC is only going to be able to measure noise from DC to the Nyquist frequency (sample rate/2). This may or may not be sufficient for your purposes, if the bandwidth of interest is included within those limits, you will get a reasonable measurement, if not, you may need to use a faster sample rate. There are other considerations as well that have to do with the data buffer depth and the frequency-domain bin size that are interelated and will have a bearing on the accuracy of your measurement.
To get a fully accurate picture of the noise and distortion of your system, you will also need to measure the THD+Noise performance of your system. You do this by injecting a clean, undistorted sine-wave into your system at the frequency of interest and then transform the time-domain sample buffer into a frequency-domain representation. From there, you can calculate the THD.
You can get good information on how to do that here:
http://en.wikipedia.org/wiki/Total_harmonic_distortion
and here:
http://www.dogstar.dantimax.dk/tubestuf/thdconv.htm
There is a very good discussion of what each of these terms mean in reference to ADC systems and the formulas for deriving them here:
http://www.analog.com/en/content/0,2886,760%255F788%255F91250,00.html
Noise is a very complex subject, the following page will get you started on understanding the various types of noise, their sources and their characteristics:
http://en.wikipedia.org/wiki/Noise_%28physics%29
Bottom line is that what you want to do isn't as simple as you may wish it to be. The pages referenced above will get you a good start on understanding the problem and how to calculate what you need.
