Well it all depends on what you are inputting to the conversion function. The definition is, given two input values a and b (that often represent a complex numbe a+j*b), magnitude = sqrt(a^2 + b^2). If a+j*b represent the complex "amplitude" of a spectrum, then the result will also be an amplitude (V). But most likely (the normal case), you are inputting a complex magnitude that will be returned as a magnitude in Vrms.