Basically my suggestions are included in the help file for 'Set Operation Mode', the error description from the error code says that the function was attempted at an illegal time.
1.) I would suggest that you make sure all move is complete before calling this function.
2.) If the axis is part of a vector space all axes should be configured with the same operation mode before uploading values.
3.) Upload the position, velocity, etc., parameters after you configure the operation mode.
If you are homing, maybe you should set all your initialization settings in MAX and then when entering your program Initialize the controller first, wait, then call the Home function. After it has finished completely then you can change the operation mode as needed.
Nestor.
Nestor