If you are able to detect a change in the CTS or DTR lines, then you could wire the output of the property node you are using to a case statement as shown below. The device connected to your serial port would have to be capable of controlling these lines. I don't understand your statement about the RTS line. This is an output from the computer. If you want to monitor the RTS line of the connected device, then you should be monitoring CTS and you will also need the correct cable. 
 
I'm not sure what your problem is with the read buffer. Does VISA Bytes at Serial Port say there is data there? If it does, you should be able to read that number of bytes. You could only directly convert that to a number if the data in the buffer represents a number (hex or ASCII). 
 
Maybe if you could explain what the RC on/off actually does. Does is it change the control lines or change the data being sent?