I'm not entirely sure if it's possible to have it both ways.
Allow the server to maintain the established connection between each client talking to each other, but to also establish a direct connection between the clients for sending of messages.
That would permit the server to run that connection timer and send instructions to each client ordering them to terminate the connection to each other. And also for the clients to have that close-to-real-time communication you want.
But it requires each client to run two threads, one listening to the server and the other for talking to the other client.