Greetings, Mathieu,
The 3458A is capable of taking up to 100,000 readings per second, depending upon how the instrument is configured. Setting MFORMAT and OFORMAT to the same setting so that no conversion is required will speed up the 3458A's ability to transfer data to GPIB. (Reading rates are fastest with MFORMAT/OFORMAT set to SINT or DINT.) You can also store the readings into the 3458A's memory so that you are not waiting for the measurement time of the 3458A to interfere with the transfer rate.
I hope this helps you separate the problems between the 3458A and the rest of your system.