how process pair redundancy can be achieved?

[srinivas88]

I am trying to implement process pairs redundancy (process level redundancy) in c/c++.
basically there is a active core process 1 running on one system and its standby copy, process 2, is running on other system.

When ever the active one goes down the standby becomes active and executes the rest of the processing.Mean while the process 1 which went down rebootes and it now behaves as the standby process.

Now there is another process 3 on another system which is sending some (TCP) messages to the active process (may be process 1 or 2 depending which is currently the active one) . One approach to process redundancy is that process 3 will be working as if there is only one process receiving/ sending back messages to it.

What i want to know is how that can be achieved. Process 3 will be given the ip address and port no. for the active process. Since the active process can be either process 1 or 2 (which are on diff. systems with different ip address/port no's) .

Is it possible to switch the ip address of process 1 or 2 depending on it is active or not?? The actual problem is that of bigger systems involving lot of processing which will involve a remote process (3) sending it messages on what set of processing to do next. I just want to know how it is done at the basic level so that i can apply it to the bigger level. May be a simple message sending and receiving programs between processes 1,2 to 3 .

Any help on the approach, possible solutions by which we can implement process pairs redundancy or even some sample progs will be greatly appreciated. The main issue here is of connectivity rather than how the messages are sent/received.

[L7Sqr]

This depends on how you manage the reboot and sending of messages. For instance, using TCP, after a reboot you will need to reconnect do to the nature of the stream connection. If the server is OK with this and can take connection requests from clients then simply have the following setup:

Server:
   Maintains a message queue to send to clients
   Listens for incoming connection requests from clients
   For each client connected, attach it to the message queue
   For each message, send to all attached clients
Client:
   On reboot, connect to server
   Receive message:
      If active: process according to message, reply to server
      If not active: discard message

The drawback to this approach is the additional bandwidth consumption on the link. If all messages are going to all clients over a constrained shared link then as you scale you will eventually flood the link. To mitigate that scenario you could add the additional logic to the server:

Server:
   listen for client connections (with MAX)
   client connects:
      if maximum client count reached: deny connection
      else: accept as above

This means that there is always at least one additional client receiving messages (and ready to become active) but no more than MAX. Passive clients (previously denied a connection) can periodically try to connect to the server to always ensure that there is a constant pool of available clients (in the face of shutdowns)

[srinivas88]

Actually here a warm stand by is used ... that is at any point there will be only 1 process receiving the messages , later after it enters another state it sends a message to the standby to update its state ... this way there is no un necessary link flooding due to the standby process ...
May be one way of doing this can be that process 3 will be given a logical ip to which it will keep on sending messages and it wouldnt be aware that a switch over has happened .. how can we give a single logical ip to the group of processes 1 and 2 .. so that process 3 keeps on sending messages without being aware of a switch over happening (if any)..

[L7Sqr]

With TCP, on separate machines, you can't. TCP is stateful and requires setup and shutdown semantics be preserved; you can not just 'plug in' to the middle of a conversation. My suggestion (or another similar approach) is as close as you are going to come. Is there any reason you cant always be sending to two clients? I dont know the characteristics of your environment.

If you can switch to UDP you could use a broadcast address and any client that becomes active just subscribes to that broadcast address and will always get all messages. However, you then run the risk of failed messages on the link. Again, I don't know enough about your environment to comment on how valid this approach is.