What is the sample rate of the signal you input into the FM modulator? The output of the FM modulator is an IQ signal at the same sample rate as the input signal.
So, you should set the IQ rate to the same sample rate as your pre-modulated signal. Often the IQ signal will be a cluster or a waveform data type that included the wavefom's IQ rate (in the form of dt between samples). The RFSG write VI has a 'Use dt as IQ Rate?' input. If you set this to true, the IQ Rate will be set to this waveform's IQ rate, and there's nothing else to do.
As far as waveform quantum, this is not something you set, but a requirement of the hardware that you have to satisfy. If you are only playing the waveform through once, you can simply drop a few samples at the end so that the total number of samples is a multiple of the quantum. If you want to repeat the waveform continuously, there are lots of ways to satisfy this. Change the waveform duration, adjust the rate a little, or resample if neither of these are options. The simplest fix is probably to use rfsg_Resample and Write.vi. This will satisfy the waveform quantum requirement by repeating the waveform. Find it in RFSG's waveform palette.
Signal bandwidth is not terribly important. You should be able to figure out the exact bandwidth of the signal you are generating, but if you just want to overestimate and use 1 MHz, that would be fine.
You can leave pre-filter gain at the default (0.0 dB). This is useful to avoid overflow, but I don't expect you'd have trouble with overflow for the waveform you are using.
