I'm not sure what you mean by the "extension" can you give more explaination of what you mean?
Concerning the Mac. They are wonderful machines, but some older machines didn't have RS232 ports. They had RS422 ports. I don't know if the newer Macs are the same, but that's something to check. RS422 ports are not necessarily inter-operable with RS232 devices.
Next make sure that you are initializing the serial port correctly (baud rate, stop bits, etc.) and have the cable wired correctly. Mac documentation will tell you the pinout of the computer, and the stepping motor control should give you the pinout of the motor.
Finally, who manufacturers the stepping motor you are using?
Mike...