One solution for you might be to use UDP instead of TCP. If you do that you can listen on a single port and get data from all your connections. Instead of tracking connections, whenever you receive a packet you also keep track of the IP address that sent it (this is one of the outputs of UDP Read), and use that address to send the response.
To do this with TCP/IP, take a look at the Multiple Connections - Server example, in the example finder under Networking -> TCP & UDP. It demonstrates one way to keep track of multiple TCP connections, although it doesn't deal with all the complexities. I once wrote code that handled this neatly but no longer have access to it to provide as an example; I can explain how it worked. Whenever any socket received data, it was converted to a variant and the connection ID was attached as an attribute (a cluster would work equally well here), then passed off to another loop for processing. When the processing loop completed it put the result into a queue, again as a variant with the same connection ID as an attribute. The sending loop retrieved the connection ID and sent the response. The sending and receiving loops were completely separate.
One complexity that is not handled in the example is how to handle connections that time out. You need to loop through each connection ID fairly quickly, checking for new data, if you want it to be responsive. It's quite likely that sometimes there will be no new data on one of the connections, generating an error that you'll need to ignore. However, you do want a way to close a connection if there's no data for a sufficiently long period of time. I handled this by putting each connection into a cluster that also contained a timestamp of the last packet received on that connection. Each time through the loop, if there was no new data and the time elapsed since the timestamp was greater than my timeout limit, I closed the connection and removed it from the array.
