Jetty Logo
Version: 9.3.28.v20191105
Contact the core Jetty developers at www.webtide.com

private support for your internal/customer projects ... custom extensions and distributions ... versioned snapshots for indefinite support ... scalability guidance for your apps and Ajax/Comet projects ... development services for sponsored feature development

Session Clustering with Infinispan

Configuration
The Infinispan Module

Jetty can support session clustering by persisting sessions to Infinispan. Each Jetty instance locally caches sessions for which it has received requests, writing any changes to the session through to Infinispan as the request exits the server. Sessions must obey the Serialization contract, and servlets must call the Session.setAttribute() method to ensure that changes are persisted.

The persistent session mechanism works in conjunction with a load balancer that supports stickiness. Stickiness can be based on various data items, such as source IP address or characteristics of the session ID or a load-balancer specific mechanism. For those load balancers that examine the session ID, the Jetty persistent session mechanism appends a node ID to the session ID, which can be used for routing.

Configuration

There are two components to session management in Jetty: a session ID manager and a session manager.

  • The session ID manager ensures that session IDs are unique across all webapps hosted on a Jetty instance, and thus there can only be one session ID manager per Jetty instance.
  • The session manager handles the session lifecycle (create/update/invalidate/expire) on behalf of a web application, so there is one session manager per web application instance.

These managers also cooperate and collaborate with the org.eclipse.jetty.server.session.SessionHandler to enable cross-context dispatch.

The Infinispan Module

When using the jetty distribution, to enable Infinispan session persistence, you will first need to enable the Infinispan module for your base using the --add-to-start or --add-to-startd argument to the start.jar.

As part of the module installation, the necessary Infinispan jars will be dynamically downloaded and installed to your ${jetty.base}/lib/infinispan directory. If you need to up or downgrade the version of the Infinispan jars, then you can delete the jars that were automatically installed and replace them. Once you’ve done that, you will need to prevent Jetty’s startup checks from detecting the missing jars. To do that, you can use --skip-file-validation=infinispan argument to start.jar on the command line, or place that line inside ${jetty.base}/start.ini to ensure it is used for every start.

You will also find the following properties, either in your base’s start.d/infinispan.ini file or appended to your start.ini, depending on how you enabled the module:

## Unique identifier for this node in the cluster
jetty.infinispanSession.workerName=node1
jetty.infinispanSession.workerName
The name that uniquely identifies this node in the cluster. This value will also be used by the sticky load balancer to identify the node. Don’t forget to change the value of this property on each node on which you enable Infinispan session clustering.

These properties are applied to the InfinispanSessionIdManager described below.

Configuring the InfinispanSessionIdManager

The Infinispan module will have installed file called $\{jetty.home}/etc/jetty-infinispan.xml. This file configures an instance of the InfinispanSessionIdManager that will be shared across all webapps deployed on that server. It looks like this:

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_3.dtd">

<Configure id="Server" class="org.eclipse.jetty.server.Server">

  <!-- ===================================================================== -->
  <!-- Get a reference to the default local cache.                           -->
  <!-- ===================================================================== -->
  <New id="local" class="org.infinispan.manager.DefaultCacheManager">
     <Get id="cache" name="cache"></Get>
  </New>

  <!-- ===================================================================== -->
  <!-- Get a reference to a hotrod client for a remote cache.               -->
  <!-- Change the name of the cache to match your setup.                    -->
  <!-- ===================================================================== -->
  <!--
  <New id="hotrodMgr" class="org.infinispan.client.hotrod.RemoteCacheManager">
    <Call id="cache" name="getCache">
      <Arg>sessions</Arg>
    </Call>
  </New>
  -->


  <!-- ===================================================================== -->
  <!-- Configure a SessionIdManager with the cache selected above.          -->
  <!-- ===================================================================== -->
  <Set name="sessionIdManager">
    <New id="idMgr" class="org.eclipse.jetty.session.infinispan.InfinispanSessionIdManager">
      <Arg>
        <Ref refid="Server"/>
      </Arg>
      <Set name="workerName"><Property name="jetty.infinispanSession.workerName" default="node1"/></Set>
      <Set name="cache"><Ref refid="cache"/></Set>
    </New>
  </Set>

</Configure>

As you can see, you configure the Infinispan Cache instance that the InfinispanSessionIdManager should use in this file. By default, the Infinispan Default cache instance is used (e.g. on the local node). You can instead use a custom Cache setup - the jetty-infinispan.xml file shows you how to configure a remote Cache (using the hotrod java client).

The InfinispanSessionIdManager can be configured by calling setters:

idleExpiryMultiple
Sessions that are not immortal, e.g. they have an expiry time, have their ids stored into Infinispan with an idle expiry timeout equivalent to double the session’s timeout. This should be sufficient to ensure that a session id that is in-use by a session is never accidentally removed. However, should you wish to, you can configure this to any integral value to effectively increase the idle expiry timeout.

Configuring the InfinispanSessionManager

As mentioned elsewhere, there should be one InfinispanSessionManager per context (e.g. webapp). It will need to reference the single InfinispanSessionIdManager configured previously for the Server.

The way you configure a InfinispanSessionManager depends on whether you’re configuring from a context xml file, a jetty-web.xml file or code. The basic difference is how you get a reference to the Jetty org.eclipse.jetty.server.Server instance.

From a context xml file, you reference the Server instance as a Ref:

  <!-- Expose the jetty infinispan classes for session serialization -->
  <Call name="prependServerClass">
    <Arg>-org.eclipse.jetty.session.infinispan.</Arg>
  </Call>


  <!-- Get a reference to the InfinispanSessionIdManager -->
  <Ref id="Server">
    <Call id="idMgr" name="getSessionIdManager"/>
  </Ref>

  <!-- Get a referencee to the Cache from the InfinispanSessionIdManager -->
  <Ref id="idMgr">
    <Get id="cache" name="cache"/>
  </Ref>

  <!-- Use the InfinispanSessionIdManager and Cache to setup up the InfinispanSessionManager -->
  <Set name="sessionHandler">
    <New class="org.eclipse.jetty.server.session.SessionHandler">
      <Arg>
        <New id="mgr" class="org.eclipse.jetty.session.infinispan.InfinispanSessionManager">
          <Set name="sessionIdManager">
            <Ref id="idMgr"/>
          </Set>
          <Set name="cache">
            <Ref id="cache">
            </Ref>
          </Set>
          <Set name="scavengeInterval">60</Set>
        </New>
      </Arg>
    </New>
  </Set>

From a WEB-INF/jetty-web.xml file, you can reference the Server instance directly:

  <!-- Expose the jetty infinispan classes for session serialization -->
  <Call name="prependServerClass">
    <Arg>-org.eclipse.jetty.session.infinispan.</Arg>
  </Call>

<!-- Reference the server directly -->
<Get name="server">
  <Get id="idMgr" name="sessionIdManager"/>
</Get>

<!-- Get a reference to the Cache via the InfinispanSessionIdManager -->
 <Ref id="idMgr">
    <Get id="cache" name="cache"/>
  </Ref>

<!-- Apply the SessionIdManager and Cache to the InfinispanSessionManager -->
<Set name="sessionHandler">
  <New class="org.eclipse.jetty.server.session.SessionHandler">
     <Arg>
        <New id="mgr" class="org.eclipse.jetty.session.infinispan.InfinispanSessionManager">
          <Set name="sessionIdManager">
            <Ref id="idMgr"/>
          </Set>
          <Set name="cache">
            <Ref id="cache">
            </Ref>
          </Set>
          <Set name="scavengeInterval">600</Set>
        </New>
      </Arg>
  </New>
</Set>

The InfinispanSessionManager can be provided by calling setters:

scavengeInterval
Time in seconds between runs of a scavenger task that looks for expired old sessions to delete. The default is 10 minutes.
staleIntervalSec
The length of time a session can be in memory without being checked against the cluster. A value of 0 indicates that the session is never checked against the cluster - the current node is considered to be the master for the session.

Using HotRod

If you’re using the hotrod client - where serialization will be required - you will need to ensure that the hotrod marshalling software works with Jetty classloading. To do this, firstly ensure that you have included the lines containing the prependServerClass to your context xml file as shown above.

Then, create the file ${jetty.base}/resources/hotrod-client.properties. Add the following line to this file:

infinispan.client.hotrod.marshaller=org.eclipse.jetty.session.infinispan.WebAppMarshaller

See an error or something missing? Contribute to this documentation at Github!(Generated: 2019-11-05)