Rolling upgrade with MDBs

We have a Java EE application deployed on a Glassfish 3.1.2 cluster which provides a REST API using JAX-RS. We regularly deploy new versions of the application by deploying an EAR to a duplicate cluster instance, then update the HTTP load balancer to send traffic to the updated instance instead of the old one.

This allows us to upgrade with no loss of availability as described here: http://docs.oracle.com/cd/E18930_01/html/821-2426/abdio.html#abdip. We are frequently making significant changes to the application, which makes the new versions "incompatible" (which is why we use two clusters).

We now have to provide a message-queue interface to the application for some high throughput internal messaging (from C++ producers). However, using Message Driven Beans I cannot see how it is possible to upgrade an application without any service disruption?

The options I have investigated are:

Single remote JMS queue (openMQ)

Producers send messages to a single message queue, messages are handled by a MDB. When we start a second cluster instance, messages should be load-balanced to the upgraded cluster, but when we stop the "old" cluster, outstanding transactions will be lost.

I considered using JMX to disable producers/consumers to that message queue during the upgrade, but that only pauses message delivery. Oustanding messages will still be lost when we disable the old cluster (I think?).

I also considered ditching the @MessageDriven annotation and creating a MessageConsumer manually. This does seem to work, but the MessageConsumer cannot then access other EJB's using the EJB annotation (as far as I know):

// Singleton bean with start()/stop() functions that 
// enable/disable message consumption

@Singleton
@Startup
public class ServerControl {

private boolean running=false;

@Resource(lookup = "jms/TopicConnectionFactory")
private TopicConnectionFactory topicConnectionFactory;

@Resource(lookup = "jms/MyTopic")
private Topic topic;
private Connection connection;
private Session session;
private MessageConsumer consumer;    

public ServerControl()
{
    this.running = false;         
}

public void start() throws JMSException {
    if( this.running ) return;

    connection = topicConnectionFactory.createConnection();
    session = dbUpdatesConnection.createSession(false, Session.DUPS_OK_ACKNOWLEDGE);
    consumer = dbUpdatesSession.createConsumer(topic);
    consumer.setMessageListener(new MessageHandler());    


    // Start the message queue handlers
    connection.start();

    this.running = true;
}

public void stop() throws JMSException {
    if( this.running == false ) return;

    // Stop the message queue handlers

    consumer.close();

    this.running = false;
}
}

// MessageListener has to invoke functions defined in other EJB's

@Stateless
public class MessageHandler implements MessageListener {

@EJB
SomeEjb someEjb; // This is null

public MessageHandler() {
}

@Override
public void onMessage(Message message) {
    // This works but someEjb is null unless I 
    // use the @MessageDriven annotation, but then I 
    // can't gracefully disconnect from the queue
}

}

Local/Embedded JMS queue for each cluster

  • Clients would have to connect to two different message queue brokers (one for each cluster).
  • Clients would have to be notified that a cluster instance is going down and stop sending messages to that queues on that broker.
  • Generally much less convenient and tidy than the existing http solution.

Alternative message queue providers

  • Connect Glassfish up to a different type of message queue or different vendor (e.g. Apache OpenMQ), perhaps one of these has the ability to balance traffic away from a particular set of consumers?

I have assumed that just disabling the application will just "kill" any outstanding transactions. If disabling the application allows existing transactions to complete then I could just do that after bringing the second cluster up.

Any help would be appreciated! Thanks in advance.

Answers


If you use highly availability then all of your messages for the cluster will be stored in a single data store as opposed to the local data store on each instance. You could then configure both clusters to use the same store. Then when shutting down the old and spinning up the new you have access to all the messages.

This is a good video that helps to explain high availability jms for glassfish.


I don't understand your assumption that when we stop the "old" cluster, outstanding transactions will be lost. The MDBs will be allowed to finish their message processing before the application stops and any unacknowledged messages will be handled by the "new" cluster.

If the loadbalancing between old and new version is an issue, I would put MDBs into separate .ear and stop old MDBs as soon as new MDBs is online, or even before that, if your use case allows for delay in message processing until the new version is deployed.


Need Your Help

asp.net inserting image into database via ListView Control

c# asp.net listview

I can insert images into database or display them in the Image Control and so on but I would like to insert them into database via ListView. I achieved that partially. During I insert a new item, I

Bamboo Deployments using MSDeploy

msdeploy bamboo

I have created a CI with bamboo for a MVC Project. Build works fine but the problem is with the deployment using msdeploy. Below is the configuration of the deployment in Bamboo