CRX Clustering

CRX provides built-in clustering capability that enables multiple repository instances running on separate machines to be joined together to act as a single repository. The repository instances within a cluster are referred to as cluster nodes (not to be confused with the unrelated concept of the JCR node, which is the unit of storage within a repository).

Each node of the cluster is a complete, fully functioning CRX instance with a distinct address on the network. Any operation (read or write) can be addressed to any cluster node.

When a write operation updates the data in one cluster node, that change is immediately and automatically applied to all other nodes in the cluster. This guarantees that any subsequent read from any cluster node will return the same updated result.

A CRX cluster consists of one master instance and some number of slave instances. A single standalone CRX instance is simply a master instance with zero slave instances.


Typically, each instance runs on a distinct physical machine. The instances must all be able to communicate directly with each other over TCP/IP.

Each cluster instance retains an independent identity on the network, meaning that each can be written to and read from independently. However, whenever a write operation is received by a slave instance, it is redirected to the master instance, which makes the change to its own content while also guaranteeing that the same change is made to all slave instances, ensuring synchronization of content.

In contrast, when a read request is received by any cluster instance (master or slave) that instance serves the request immediately and directly. Since the content of each instance is synchronized by the clustering mechanism, the results will be consistent regardless of which particular instance is read from.

The usual way to take advantage of this is to have a load-balancing webserver (for example, the Dispatcher) mediate read requests and distribute them across the cluster instances, thus increasing read responsiveness.

When setting up a clustered installation, there are two main deployment decisions to be made:

• How the cluster fits into the larger architecture of the installation
• How the cluster is configured internally

The cluster architecture, on the other hand, involves how the cluster is used within the installation. The two main options are:

• Active/active clustering
• Active/passive clustering

The cluster configuration defines which internal storage mechanisms are used and how these are shared or synchronized across cluster nodes. The three most commonly used configurations are:

• Shared nothing
  
• Shared data store
  
• Shared external database
  

First we will look at cluster architecture. Cluster configuration will be covered later on.

Because CQ is built on top of the CRX repository, CRX clustering can be employed in CQ installations to improve performance. To understand the various ways that CRX clustering can fit into the larger CQ architecture we will first take a look at two common, non-clustered CQ architectures: single publish and multiple publish.

A CQ installation consists of two separate environments: author and publish. 

The author environment is used for adding, deleting and editing pages of the website. When a page is ready to go live, it is activated, causing the system to replicate the page to the publish environment from where it is served to the viewing audience.

In the simplest case, the author environment consists of a single CQ instance running on a single server machine and the publish instance consists of another single CQ instance running on another machine. In addition, a Dispatcher is usually installed between the publish server and the web for caching.

A common variation on the installation described above is to install multiple publish instances (usually on separate machines). When a change made on the author instance is ready to go live, it is replicated simultaneously to all the publish instances.

As long as all activations and deactivations of pages are performed identically on all publish instances, the content of the publish instances will stay in sync. Depending on the configuration of the front-end server, requests from web surfers are dealt with in one of two ways:

• The incoming requests are distributed among the publish instances by the load-balancing feature of the Dispatcher.
• The incoming requests are all forwarded to a primary publish instance until it fails, at which point all requests are forward to the secondary publish instance (in this arrangement there are usually only two instances).

Clustering architectures usually fall into one of two categories: active/active or active/passive.

In an active/active setup the processing load is divided equally among all nodes in the cluster using the a load balancer such as the CQ Dispatcher. In normal circumstances, all nodes in the cluster are active and running at the same time. When a node fails, the load balancer redirects the requests from that node across the remaining nodes.

In an active/passive setup, a front-end server passes all requests back to only one of the cluster nodes (called the primary) which then serves all of these requests itself. A secondary node (there are usually just two) runs on standby. Because it is part of the cluster the secondary node remains in sync with the primary, it just does not actually serve any requests itself. However, if the primary node fails then the front-end server detects this and a "failover" occurs where the server then redirects all requests to the secondary node instead.

The Active/Passive setup described above can also be thought of and used as a "hot backup" system. The secondary server (the slave) functions as the backup for the primary server. Because the clustering system automatically synchronizes the two servers, no manual backing-up is required. In the case of the failure of the primary server, the backup can be immediately deployed by activating the secondary connection.

Above we discussed two common CQ architectures that do not use CRX clustering and then gave two general examples of how CRX clsuterng can work (active/active and active/passive). Now we will bput these together and see how CRX clustering can be used within a CQ installation.

There are a number of options for using true CRX clustering within a CQ installation. The first one we will look at is publish clustering.

In the publish clustering arrangement, the publish instances of the CQ installation are combined into a single cluster. The front-end server (including the dispatcher) is then configured either for active/active behavior (where load is equally distributed across cluster nodes) or active/passive behavior (where load is only redirected on failover).

The following diagram illiustrates a publish cluster arrangement:

Clustering can also be employed to improve the performance of the CQ author environment. An arrangement using both publish and author clustering is shown below:

It is also possible to set up other variations on the author clustering theme by pairing the author cluster with either multiple (non-clustered) publish instances or with a single publish instance. However, such variants are rarely used in practice.

When faced with a performance problem in single instance CQ system (either at the publish or author levels), clustering may provide a solution. However, the extent of the improvement, if any, depends upon where the performance bottleneck is located.

Under CQ/CRX clustering, read performance scales linearly with the number of nodes in the cluster. However, additional cluster nodes will not increase write performance, since all writes are serialized through the master node.

Clearly, the increase in read performance will benefit the publish environment, since it is primarily a read system. Perhaps surprisingly, clustering can also benefit the author environment because even in the author environment the vast majority of interactions with the repository are reads. In the usual case 97% of repository requests in an author environment are reads, while only 3% are writes.

This means that despite the fact that CQ clustering does not scale writes, clustering can still be a very effective way of improving performance both at the publish and author level.

However, while increasing the number of cluster nodes will increase read performance, a bottleneck will still be reached when the frequency of requests gets to the point where the 3% of requests that are writes overwhelm the capabilities of the single master node.

In addition, while the percentage of writes under normal authoring conditions is about 3%, this can rise in situations where the authoring system handles a large number of more complex processes like workflows and multisite management.

In cases where a write bottleneck is reached, additional cluster nodes will not improve performance. In such circumstances the correct strategy is to increase the hardware performance through improved CPU speed and increased memory.

Clustering in CRX can be configured in a number of ways, depending on the implementation chosen for each element of the storage system. To understand the options available, a brief overview of CRX storage may therefore be helpful.

Storage in CRX is made up of the following major elements:

• Persistence store
• Data store
• Journal
• Version storage
• Other file-based storage

The repository's primary content store holds the hierarchy of JCR nodes and properties. This storage is handled by a persistence manager (PM) and is therefore commonly called the persistence store (despite the fact that all the stoage elements are in fact persistent). It is also sometimes referred to as the workspace store because it is configured per workspace (see below). 

While a number of different PMs are available, each using a different underlying storage mechanism, the default PM is the Tar PM, a high-performance database built into CRX and designed specifically for JCR content. It derives its name from the fact that it stores its data in the form of standard Unix-style tar files in the file system of the server on which you are running CRX. Other PMs, such as the MySQL PM and the Oracle PM, store data in conventional relational databases, which must be installed and configured separately.

All non-binary property values and all binary properrty values under a certain (configurable) size, are stored directly by the PM in the content hierarchy, in the manner specific to that PM. However, binary property values above the threshold size are redirected to the data store, and a reference to the value in the data store is stored by the PM (DS, see below).

Depending on the cluster configuration, each cluster node may have its own separate PM storage (the shared nothing and shared data store configurations) or it may share the same PM storage as the other cluster nodes (the shared external database arrangement). In the case where each cluster node has its own separate PM storage, these are kept synchronized across instances through the journal (see below). For more details on persistence managers, see Persistence Managers.

The data store (DS) holds binary property values over a given, configurable, size. On write, these values are streamed directly to the DS and only an reference to the value is written by the PM to the persistence store. By providing this level of indirection, the DS ensures that large binaries are only stored once, even if they appear in multiple locations within a workspace. In a clustered environment the data store can be either per repository instance (per cluster node) or shared among cluster nodes in commonly accessible network file system directory. For more details on the data store, see Data Store.

Whenever the repository writes data it first records the intended change in the journal. Maintaining the journal helps ensure data consistency and helps the system to recover quickly from crashes. In a clustered environment the journal plays the critical role of synchronizing content across cluster instances.

The version storage is held in a single, per-repository, hidden workspace only accessible through the versioning API. Though the version storage is reflected into each normal workspace under/jcr:system/jcr:versionStorage, the actual version information resides in the hidden workspace.

Since the version information is stored in the same way as content in a normal workspace, any changes to it in one cluster instance are propagated to the other instances in the same way as changes in a normal workspace. 

The PM used for version storage is configured separately from those used for normal workspaces. By default the Tar PM is used.

In addition to the above elements, CRX also stores some repository state information in plain files in the file-system. This includes the search indices, namespace registry, node type registry and access control settings. In a cluster, this information is automatically synchronized across cluster nodes.

As mentioned above, there are three commonly used cluster configurations, which differ according to how the various storage elements are configured. The configuration parameters are found in the file /crx-quickstart/repository/repository.xml.

This is the default configuration. In this configuration all elements of CRX storage are held per cluster node and synchronized over the network. No shared storage is used. The TarPersistenceManager is used for the persistence store and version storage, the TarJournal is used for the journal and the ClusterDataStore is used for the data store.

In most cases you will not need to manually configure the repository.xml file for shared nothing cluster, since the GUI clustering setup automatically takes care of it (see GUI Setup of Shared Nothing Clustering).

In this configuration the workspace stores and the journal are maintained per-cluster node as above, using the TarPersistenceManager and TarJournal, but the DataStore element is configured with the class FileDataStore (instead of ClusterDataStore as above). The parameter path points to the location on a shared file system where the data store will be held. Every node in the cluster must be configured to point to the same shared location.

In this configuration the workspace stores is maintained per-cluster node as above, using the TarPersistenceManager, but the Data Store uses the class FileDataStore (instead of ClusterDataStore as above) and the Journal uses the class FileJournal (instead of TarJournal). The resulting configuration for the data store is the same as immediately above. The configuration for the Journal is as follows:

• syncDelay: By default, a cluster instance reads the journal and updates its state (including its cache) every 5000 milliseconds (5 seconds). To use a different value, set the attribute syncDelay in the Cluster element. Note: This attribute belongs directly in the Cluster element. It is not a param element within the contained Journal element.
• sharedPath: Mandatory argument specifying the shared directory. 
• maximumSize: Maximum size of a single journal file before it will get rotated. Default: 104857600 bytes (100 MB).
• maximumAge: Age specified as duration in ISO 8601 or plain format. Journal files exceeding this age will get removed. If this parameter is not specified, this option is disabled (age is not checked). If the parameter is set to an empty string (""), only one Journal log file is kept. The default in CRX is "P1M", which means files older than one month are deleted. 
• maximumFiles: Number of rotated journal files kept around. Files exceeding this limit will be removed. A value of 0 disables this option. Default: 0. 
• portList: The list of ports to use in a master node. By default, any free port is used. When using a firewall, open ports must be listed. A list of ports or ranges is supported, for example: 9100-9110 or 9100-9110,9210-9220. Default: 0 (any port). 
• bindAddress: Set this parameter if synchronizing among cluster nodes should be done over a specific network interface. By default, all network interfaces are used. Default: empty (use all interfaces).
• connectTimeout: Timeout in milliseconds that a client will wait when initially connecting to a server instance before aborting. Default: 2000. 
• socketTimeout: Socket timeout in milliseconds for both reading and writing over a connection from a client to a server. After this timeout period has elapsed with no activity, the connection is dropped. Default: 60000 (1 minute). If you have very long transactions, use 600000 (10 minutes).
• operationTimeout: Operation timeout in milliseconds. After a client or server has locked the journal and this timeout has passed with no record being appended, the lock to the journal is automatically released. Default: 30000 (30 seconds). If you have very long transactions, use 600000 (10 minutes).

In this configuration all cluster nodes shared a common persistence store, journal and data store, which are all held in a single external RDBMS.

In addition, the external database storage is sometimes used to store the version storage as well (again, a single store shared across all cluster nodes).

When configuring CRX to use an external database, typically, a single backend database system is used to store all the elements (Workspaces, Data Store, Journal and Version Storage) for all instances in the cluster. Note that nothing prevents this backend database system from itself being a clustered system.

This section describes the details of cluster configuration and serves as a general guide to all possible variations on clustering. Not all possible combinations are recommended or commonly used. See the Common Configurations.

The configuration parameters for clustering are found primarly in the the file crx-quickstart/repository/repository.xml.

We will examine those sections most relevant to cluster configuration, highlighted below.

The data store is used to store property values above a given configurable size threshold. The persistence manager directly stores only the node/property hierarchy and the smaller property values while using references to the data store to represent the larger property values in the hierarchy. The data store has the following features:

• It is space saving. Only one copy per unique object is kept. For example, if two or more identical large binaries appears in the content hierarchy, only one copy will actually be stored in the data store and referenced from multiple locations.
• Copying is fast. Only the identifier is copied.
• Storing and reading does not block others.
• Multiple repositories can use the same data store.
• Objects in the data store are immutable.
• Garbage collection is used to purge unused objects.
• Hot backup is supported.

The ClusterDataStore is used with the TarPersistenceManager and the TarJournal to form a Shared Nothing cluster. Each cluster node has its own separate data store, persistence store and journal.

• minRecordLength: The minimum size for an object to be stored in the data store as opposed to the inline within the regular PM. The default is 4096 bytes. The maximum supported value is 32000.

The FileDataStore is usually used with the TarPersistenceManager and the TarJournal to configure a Shared Data Store cluster. In this configuration the persistence store and journal are still per-cluster node, but the data store is shared across the nodes. 

This data store places each binary in a separate file. The file name is the hash code of the content. When reading, the data is streamed directly from the file. No local or temporary copy of the file is created. New content is first stored in a temporary file, and later renamed and moved to the right place.

Because the data store is append-only, the FileDataStore is guaranteed to be consistent after a crash. It is usually faster than the DbDataStore, and the preferred choice unless you have strict operational reasons to put everything into a database (see below).

• path(optional): The name of the directory where this data store keeps the files. The default used if this parameter is missing is crx-quickstart/repository/repository/datastore. When configuring a Shared Data Store cluster each cluster node must be configured with this parameter pointing to a common shared filesystem location. This will typically be the mount point on the local filesystem of each machine that is bound to the common NAS storage, for example, /shared/datastore.
• minRecordLength: The minimum object length. The default is 100 bytes; smaller objects are stored inline (not in the data store). Using a low value means more objects are kept in the data store (which may result in a smaller repository, if the same object is used in many places). Using a high value results in fewer objects being stored in the data store (which may result in better performance, because less data store access is required). The maximum value is approximately 32000.

The DbDataStore is usually used with a database persistence manager (for example, MySqlPersistenceManager) and the DatabaseJournal to configure a Shared External Database cluster.

The DbDataStore stores its data in a relational database using JDBC. All content is stored in one table with the unique key of the table is the hash code of the content.

When reading, the data may be first copied to a temporary file on the server, or streamed directly from the database (depending on the copyWhenReading setting).

New content is first stored in the table under a unique temporary identifier, and later the key is updated to the hash of the content.

• url: The database URL (required).
• user: The database user name (required).
• password: The database password (required).
• databaseType: The database typeThe default is the sub-protocol of the JDBC database URL is used if it is not set. It must match the resource file [databaseType].properties. Example: mysql.properties. Currently supported are: db2, derby, h2, mssql, mysql, oracle, sqlserver.
• driver: The JDBC driver class name. By default the default driver of the configured database type is used.
• maxConnections: Set the maximum number of concurrent connections in the pool. At least 3 connections are required if the garbage collection process is used.
• copyWhenReading: Enabled (true) by default. If enabled, a stream is always copied to a temporary file when reading a stream, so that reads can be concurrent. If disabled, reads are serialized.
• tablePrefix: The table name prefix. The default is empty. Can be used to select an non-default schema or catalog. The table name is constructed as as follows: ${tablePrefix}${schemaObjectPrefix}${tableName}.
• schemaObjectPrefix: The schema object prefix. The default is empty.

A CRX repsoitory is composed of one or more worksapces, each of which holds a tree of nodes in properties. These are configured by the elements Workspaces and Workspace (see below). The Workspaces element specifies where the worksapce data is to be stored and which of the workspaces will be the default workspace:

• rootPath: The native file system directory for workspaces. A subdirectory is automatically created for each workspace, and the path of that subdirectory can be used in the workspace configuration as the {{${wsp.path}}} variable.
• defaultWorkspace: Name of the default workspace. This workspace is automatically created when the repository is first started.
• maxIdleTime:(Optional) By default CRX only releases resources associated with an opened workspace when the entire repository is closed. This option, if specified, sets the maximum number of seconds that a workspace can remain unused before the workspace is automatically closed. Changing this parameter is not recommended.
  
• configRootPath: (Optional) By default the configuration of each workspace is stored in a workspace.xml file within the workspace directory within the rootPath directory. If this option is specified, then the workspace configuration files are stored within the specified path in the virtual file system specified in the FileSystem element within the repository.xml file. Changing this parameter is not recommended.
  
  

The Workspace element serves as a template for the creation of individual workspace.xml files for each workspace created in the rootPath configured above, in the Workspaces element. The workspace configuration template and all workspace.xml configuration files have the following structure:

• name: The name of the workspace. This will be automatically filledin when the actual worksapce.xml file is created from this template.
• simpleLocking: A boolean indicating whether simple locking is used on this worksapce. The default is true.

The persistence manager is the main storage mechanism of the reposiroty. It stores the node and property hierarchy and values of each property (with the expcetion of large binary values, which are referenced from within the persitence storage and actually stored in the Data Store as discussed above).

The default persistence manager (and the one used for Shared Nothing clustering) is the TarPersistenceManager:

The persistence store can also be housed in an external database (as is is done in the Shared External Database configuration of clustering, for example). There are a number of database persistence managers available. For information about acquiring these persistence managers see the Jackrabbit project. The classes in question are all subclasses BundleDbPersistenceManager. They are:

• DerbyPersistenceManager
• H2PersistenceManager
• MSSqlPersistenceManager
• MySqlPersistenceManager
• OraclePersistenceManager
• PostgreSQLPersistenceManager

The element <Cluster> holds the parameters that govern the journal.

The default journal implementation is the TarJournal. This journal is used for both the Shared Nothing and Shared Data Store configurations.

• syncDelay: Events that were issued by other cluster nodes are processed after at most this many milliseconds. Optional, the default is 5000 (5 seconds).
• bindAddress: Used if the synchronization between cluster nodes should be done over a specific network interface. Default: empty (meaning all network interfaces are used).
• maxFileSize: The maximum file size per journal tar file. If the current data file grows larger than this number (in megabytes), a new data file is created (if the last entry in a file is very big, a data file can actually be bigger, as entries are not split among files). The maximum file size is 1024 (1 GB). Data files are kept open at runtime. The default is 256 (256 MB).
• maximumAge: Age specified as duration in ISO 8601 or plain format. Journal files that are older than the configured age are automatically deleted. The default is "P1M", which means files older than one month are deleted.
• portList: The list of listener ports to use by this cluster node. When using a firewall, the open ports must be listed. A list of ports or ranges is supported, for example: 9100-9110 or 9100-9110,9210-9220. By default, the following port list is used: 8088-8093.
• becomeMasterOnTimeout: Flag indicating whether a slave may become master when the network connection to the current master times out. Default: false. If this parameter is set to true, then the slave will attempt to contact the master up to 5 times, where the timeout for each attempt is defined by the system property socket.receiveTimeout. By default this system property is set to 60000 milliseconds (1 minute), so the default total timeout for a slave to become master is 5 minutes.

The FileJournal stores the journal data in a plain file, usually in a location shared by all cluster nodes (through I directory bound to a NAS, for example), in a manner similar to the FileDataStore (above). This class is used in the Shared Data Store and Journal configuration.

• revision: The name of this cluster node's revision file; this is a required property with no default value
• directory: The directory where the journal file as well as the rotated files are kept; this is a required property with no default value.
• basename: The basename of the journal files; the default value is DEFAULT_BASENAME. It is recommended that this setting not be changed.
• maximumSize: The maximum size of an active journal file before rotating it: the default value is DEFAULT_MAXSIZE which is equal to 1048576. It is recomended that this setting not be changed.

The journal can also be stored in an external database. A number of database journal implementations are available. These classes are used in the Shared External Database configuration.

• With an Oracle database use OracleDatabaseJournal.
• With an MS SQL database use MSSqlDatabaseJournal.
• For other generic databases (accessible through JNDI) use the base DatabaseJournal class. Do not use the JNDIDatabaseJournal, it has been deprecated.

The file crx-quickstart/repository/cluster_node.id contains the cluster node ID. Each instance within a cluster must have a unique ID.

This file is automatically created by the system. By default it contains a randomly generated UUID, but it can be any string. When copying a cluster node, this file should be copied. If two nodes (instances) within a cluster contain the same cluster node id, only one of them will be able to connect.

Example file:

The file crx-quickstart/repository/cluster.properties contains cluster configuration properties. The file is automatically updated by the system if the cluster configuration is changed in the GUI.

Example file:

The cluster_id property contains the identifier for this cluster, each node in the cluster must have the same cluster ID. By default this is a randomly generated UUID, but it can be any string.

The addresses property contains a comma separated list of the IP addresses of all nodes in this cluster. This list is used at the startup of each cluster node to connect to the other nodes that are already running. The list is not needed if all cluster nodes are running on the same computer (which may be the case in certain circumstances, such as testing).

The members properties contains a comma separated list of the cluster node IDs that participate in the cluster. This property is not required for the cluster to work, it is for informational purposes only.

The following system properties affect the cluster behavior:

socket.connectTimeout: The maximum number of milliseconds to wait for the master to respond (default: 1000). A timeout of zero means infinite (block until connection established or an error occurs).

socket.receiveTimeout: the maximum number of milliseconds to wait for a reply from the master or slave (SO_TIMEOUT; default: 60000). A timeout of zero means infinite.

com.day.crx.core.cluster.DisableReverseHostLookup: Disable the reverse lookup from the master to the slave when connecting (default: false). If not set, the master checks if the slave is reachable using InetAddress.isReachable(connectTimeout).

In this section we describe the process of setting up a CRX cluster.

CRX clustering is designed to operate using network connectivity equivalent to that available between servers in a data centre. Optimal cluster perfomance requires very low network latency between nodes, where available bandwidth is at least 100 MBits/sec, and network links between nodes are not congested with non-cluster traffic. The most common situation, where servers are connected to a common network switch, is ideal.

Because of the need for rapid round-trip communication between cluster nodes during an update, the performance of a CRX cluster is sensitive to the latency between nodes. Where network latency can be kept in the millisecond range, cluster operation is likely to be satisfactory, even when the nodes do not share the same physical location. However, physically separate datacentres often experience network latencies that are higher than 10 ms, and this latency reduces cluster performance.

The following requirements should be observed when setting up clustering:

• Each cluster node (CRX instance) should be on its own dedicated machine. During development and testing one can install multiple instances on a single machine, but for a production environment this is not recommended.
• For shared-nothing clustering, the primary infrastructure requirement is that the netwrok connecting the cluster nodes have high reliability, high-availability and low-latency.
• For shared data store and shared external database clustering, the shared storage (be it file-based or RDBMS-based) should be hosted on a high-reliability, high-availability storage system. For file storage the recommended technologies include enterprise-class SAN systems, NFS servers and CIFS servers. For database storage a high-availability setup running either Oracle or MySQL is recommended. Note that since, in either case, the data is ultimately stored on a shared system, the reliability and availability of the cluster in general depends greatly on the reliability and availability of this shared system.

By default a freshly installed CRX instance runs as a single-master, zero-slave, shared-nothing cluster. Additional slave nodes can be added to the master easily through the cluster configuration GUI.

If you wish to deploy a shared data store or shared external database cluster, or if you wish to tweak the settings of the default shared-nothing cluster, you will have to perform a manual configuration. In this section we describe the GUI deployment of a shared-nothing cluster. Manual configuration is covered in the next section.

In some cases a user may wish to set up a cluster without using the GUI. There are two ways to do this: manual slave join and manual slave clone.

The first method, manual slave join, is the same as the standard GUI procedure except that it is done without the GUI. Using this method, when a slave is added, the content of the master is copied over to it and a new search index on the slave is built from scratch. In cases where an pre-existing instance with a large amount of content is being "clusterized" this process can take a long time.

In such cases it is recommended to use the second method, manual slave clone. In this method the master instance is copied to a new location either at the file system level (i.e., the crx-quickstart directory is copied over) or using the online backup feature and the new instance is then adjusted to play the role of slave. This avoids the rebuilding of the index and for large repositories can save a lot of time.

The following steps are similar to joining a cluster node using the GUI. That means the data is copied over the network, and the search index is re-created (which may take some time):

If a quickstart deployment is being used (i.e., stand-alone, without an application server) then:

On the master

• Copy the files crx-quickstart-*.jar and and license.properties to the desired directory.
• Start the instance:
  java -Xmx512m -jar *.jar
• Verify that the instance is up and running, then stop the instance.
• In the file crx-quickstart/repositorycluster.properties, add the IP address of the slave instance you are adding below (if that slave is on a different machine, otherwise this step is not needed).
• Stop the instance.
• If a shared data store is to be be used:
  • Change
    crx-quickstart/repository/repository.xml, as described in Shared Data Store and Data Store, above.
  • Assuming you have configured <shared>/datastore/ as the shared location, copy the contents of the crx-quickstart/repository/repository/datastore tothe directory <shared>/datastore.
• Start the instance, and verify that it still works.

On the slave

• Copy the files crx-quickstart-*.jar and license.properties to the desired directory (usually on a different machine from the master, unless you are just testing).
• Unpack the JAR file:
  java -Xmx512m -jar crx-quickstart-*.jar -unpack
• Copy the files repository.xml and cluster.properties from the master:
  cp ../n1/crx-quickstart/repository/repository.xml crx-quickstart/repository/
  cp ../n1/crx-quickstart/repository/cluster.properties crx-quickstart/repository/
• Copy the namespaces and node types from the master:
  cp -r ../n1/crx-quickstart/repository/repository/namespaces/ crx-quickstart/repository/repository/
  cp -r ../n1/crx-quickstart/repository/repository/nodetypes/ crx-quickstart/repository/repository/
• If this new slave is on a different machine from the master, append the IP address of the master to the cluster.properties file of the slave:
  echo "addresses=x.x.x.x" >> crx-quickstart/repository/cluster.properties
  where x.x.x.x is replaced by the correct address. As mentioned above, the IP address of the slave should be added to the master's cluster.properties file as well.
• Start the slave instance:
  java -Xmx512m -jar crx-quickstart-*.jar

If an application server deployment is being used (i.e., the CQ or CRX war file is being deployed) then:

On the master

• Deploy the war file into the application server. See Installing CRX in an Application Server and Installing CQ in an application server.
• Stop the application server.
• In the file crx-quickstart/repositorycluster.properties, add the IP address of the slave instance you are adding below (if that slave is on a different machine, otherwise this step is not needed).
• If a shared data store is to be be used change crx-quickstart/repository/repository.xml, as described in Shared Data Store and Data Store, above.
• Move the datastore directory to the required place
• Start the application server, and verify that it still works.
  

On the slave

• Deploy the war file into the application server.
• Stop the application server.
• Copy the files repository.xml and cluster.properties from the master:
  cp ../n1/crx-quickstart/repository/repository.xml crx-quickstart/repository/
  cp ../n1/crx-quickstart/repository/cluster.properties crx-quickstart/repository/
• Copy the namespaces and node types from the master:
  cp -r ../n1/crx-quickstart/repository/repository/namespaces/ crx-quickstart/repository/repository/
  cp -r ../n1/crx-quickstart/repository/repository/nodetypes/ crx-quickstart/repository/repository/
• If this new slave is on a different machine from the master, append the IP address of the master to the cluster.properties file of the slave:
  echo "addresses=x.x.x.x" >> crx-quickstart/repository/cluster.properties
  where x.x.x.x is replaced by the correct address. As mentioned above, the IP address of the slave should be added to the master's cluster.properties file as well.
• Start the application server.

The following steps clone the master instance and change that clone into a slave, preserving the existing search index:

Master

Your existing repository will be the master instance.

If it is feasible to stop the master instance:

• Stop the master instance either through the GUI switch, the command line stop script orr, in the case of an application server deployment, by stopping the application server.
• In the case of a quickstart deployment, copy the crx-quickstart directory of the master over to the location where you want the slave installed, using a normal filesystem copy (cp, for example).
• In the case of an application server installation, copy the exploded war file from the master to same location in the slave application server (see Installing CRX in an Application Server and Installing CQ in an application server).
• Restart the master.

If it is not feasible to stop the master instance:

• Do an online backup of the instance to the new slave location. The online backup tool can be made to write the copy directly into another directory or to a zip file which you can then unpack in the new location. See here for details. The process can be automated using curl or wget. For example:
  
  curl -c login.txt "http://localhost:7402/crx/login.jsp?UserId=admin&Password=xyz&Workspace=crx.default"
  
  curl -b login.txt -f -o progress.txt "http://localhost:7402/crx/config/backup.jsp?action=add&&zipFileName=&targetDir=<targetDir>"
   

Slave

In the new slave instance directory (or, in the application server case, the exploded war file directory of the slave):

• Modify the file crx-quickstart/repository/cluster_node.id so that it contains a unique cluster node ID. This ID must differ from the IDs of all other nodes in the cluster.
• Add the node ID of the master instance and all other slave nodes (apart from this one) separated by commas to the file crx-quickstart/repository/cluster.properties. For example, you could use something like the following command (with the capitalized items replaced with the actual IDs used):
  
  echo "members=MASTER_NODE_ID,SLAVE_NODE_1_ID" >> crx- quickstart/repository/cluster.properties
  
  
• Add the master instance IP address and the IP address of all other slave instances (apart from this one) to the file crx-quickstart/repository/cluster.properties. For example, you could use something like the following command (with the capitalized items replaced with the actual IDs used):
  
  echo "addresses=MASTER_IP_ADDRESS,SLAVE_1_IP_ADDRESS" >> crx- quickstart/repository/cluster.properties
  
  
• Remove the file sling.id.file, which stores the instance id by which nodes in a cluster are distinguished from each other. This file will get re-generated with a new ID if missing, so deleting it is sufficient.
  The file is in different places depending on installation, so it needs to be found:
  
  rm -i $(find -type f -name sling.id.file)
  
  
• Start the slave instance. It will join the cluster without re-indexing. Note: Once the slave is started, the master cluster.properties file will automatically be updated by appending the node ID and IP address of the slave.

In some cases, when the master instances is stopped while the other cluster instances are still running, The master instance cannot re-join the cluster after being restarted.

This can occur in cases where a write operation was in progress at the moment that the master node was stopped, or where a write operation occured a few seconds before the master instance was stopped. In these cases, the slave instance may not receive all changes from the master instance. When the master is then re-started, CRX will detect that it is out of sync with the remaining cluster instances and the repository will not start. Instead, an error message is written to the server.log saying the repository is not available, and the following or a similar error message in the file crx-quickstart/logs/crx/error.log and crx-quickstart/logs/stdout.log:

To avoid this problem, ensure that the slave cluster instances are always stopped before the master is stopped.

If you are not sure which cluster instance is currently the master, open the page http://localhost:port/crx/config/cluster.jsp. The master ID listed there will match the the contents of the file crx-quickstart/repository/cluster_node.id of the master cluster instance.

To avoid desynchronization of cluster nodes the following mechanism is used:

1. As soon as a cluster of two or more nodes is connected, a marker file named clustered.txt is created on each cluster node in its repository root directory (crx-quickstart/repository/clustered.txt). This includes the master node and all slave nodes.
2. When a master node detects that there are no longer any slave nodes attached to it (for example, if all slave nodes have failed) then it automatically deletes its own clustered.txt file.
3. The file is not deleted when a slave node is stopped normally. 
4. Also, the file is not deleted when any node (master or slave) is stopped abnormally (i.e., if the node process is killed or a crash occurs).

The result of this is that in a cluster that has failed, any node that was a slave previously will have this marker file while the former master will not have this file, unless it was shutdown abnormally.

If the file exists on a cluster node when it is restarted, that cluster node will only start as a slave, not a master. It will then attempt to reconnect to a master, once a second, for 60 seconds (by default, see below). If it fails to connect in that time then the startup is aborted and the node shuts down.

In order to restart the cluster in this case you must restart the master first, make sure it is up and running, and then restart the slaves.

If the file clustered.txt exists on all nodes in a cluster, including the former master, this means that there is much higher chance that the the cluster is out-of-sync. There fore the administrator must elect one of the nodes to be the new master, remove the clustered.txt from that node and clone that node to create new slave nodes (cloning is described in Manual Slave Clone, above).

The choice of which of the nodes should be the new master can be complex. In general it is best to choose the node which has the most data, since this usually indicates that it was the most recent to be updated (recall that the tarPM is append-only, so over short periods of time we can rely on the larger store being the most recently updated). To detemine this the administrator must compare the sizes of the most recent data_*.tar file in each of the following locations

• the default workspace (crx-quickstart/repository/workspaces/crx.default/data_*.tar) 
• the version workspace (crx-quickstart/repository/version/data_*.tar), and
• the journal (crx-quickstart/repository/version/data_*.tar).

Note that this method of determining which noe to make the new master is not foolproof and other factors may need to be taken into consideration.

The maximum number of seconds that a slave will attempt to connect with  a master can be changed by setting the system property

com.day.crx.core.cluster.WaitForMasterRetries (default: 60)

There is a one second delay after each attempt, so the default results in 60 attempts before giving up.

To re-join a cluster instance that is out of sync, there are a number of solutions:

• Create a new repository and join the cluster node as normal.
• Use the Online Backup feature to create a cluster node. In many cases this is the fastest way to add a cluster node.
• Restore an existing backup of the cluster instance node and start it.
• As described in the error message, delete the index and data tar files that are out-of-sync on this cluster node and restart. Note that the Lucene search index may still be out of sync unless it is also deleted. This procedure is discouraged as it requires more knowledge of the repository, and may be slower than using the online backup feature (specially if the Lucene index needs to be re-built).

The time that it takes to synchronize an out-of-sync cluster node depends on two factors:

• The length of time that the node has been disconnected from the cluster.
• The rate of change of information in the live node.

Combined, these factors determine how much data has changed during the time that the node has been disconnected, and therefore, the amount of data that needs to be transfered and written to that node to re-synchronize it.

Since the time taken depends on these variables, it will differ from installation to installation. However, by making some reailistic worst-case scenario projections we can estimate an example sync-time:

• Assume that a cluster node will not be out of sync for any more than 24 hours, since in most cases this is sufficient time for manual intervention to occur.
• Also assume that the website in question is the main corporate website for an enterprise of approximately 10,000 employees. 24 hours of work on a website in such an organization is projected as follows:
  • import 250 2MB images
  • create 2400 pages
  • perform 10000 page modifications
  • activate 2400 pages

Internal testing at Adobe has shown that given the above assumptions the results for a CQ 5 installation using CRX clustering are:

• Synchronization took 13 minutes and 38 seconds.
• The author response time increased from 400ms to 700ms during the synchronization (including. network latency).
• The inital crx-quickstart folder size was 1736MB.
• The crx-quickstart folder size after the simulated 24h was 2588MB.
• Therefore a total transfer of 852MB of was needed to perform the synchronization.

Active clusters do not support session-scoped locks. Open-scoped locks or application-side solutions for synchronizing write operations should be used instead.

The Tar PM stores its data in standard Unix-style tar files. Occassionally, you may want to optimize these storage files to increase the speed of data access. Optimizing a Tar PM clustered system is essentially identical to optimizing a stand-alone Tar PM instance (see Optimizing Tar Files).

The only difference is that to optimize a cluster, you must run the optimization process on the master instance. If the optimization is started on the master instance, the shared data as well as the local cache of the other cluster instances is optimized automatically. There is a small delay before the changes are propagated (a few seconds). If one cluster instance is not running while optimizing, the tar files of that cluster instance are automatically optimized the next time the instance is started.

In an active/passive environment with two nodes, under normal operating conditions, all incoming request are served by the master, while the slave maintains content synchronization with the master but does not itself serve any requests. Ideally, if the master node fails, the slave, having identical content to the master, would automatically jump in and start serving requests, with no noticeable downtime.

Of course, there are many cases where this ideal behavior is not possible, and a manual intervention by an adminstrator is required. The reason for this is that, in general, the slave cannot know with certainty what type of failure has occured. And the type of failure (and there many types) dictates the appropriate response. Hence, intelleigent intervention is usually required.

For example, imagine a two node active/passive cluster with master A and slave B. A and B keep track of each other's state through "heartbeat" signal. That is, they periodically ping each other and wait for a response.

Now, if B finds that its pings are going unanswered for some relatively long period, there are, generally speaking, two possible reasons:

1. A is not responding because it is inoperable.
2. A is still operating normally and the lack of response is due to some other reason, most likely a failure of the network connection between the two nodes.

If (1) is the case then the logical thing is for B to become the master and for requests to be redirected to and served by B. In this situation, if B does not take on A's former role, the service will be down; an emergency situation.

But if (2) is the case then the logical thing to do is for B to simply wait and continue pinging until the network connection is restored, at which point it will resynchronize with the master, effecting all changes that occured on the master during the down time. In this situation, if B instead assumes that A is down and takes over, there would be two functioning master nodes, in other words a "split-brain" situation where there is a possibility that the two nodes become desynchronized.

Because these two scenarios are in conflict and the slave cannot distinguish between them, the default setting in CRX is that upon repeated failures to reconnect to a non-responsive master the slave does not become the master. However, see below for a case where you may wish alter this default behavior.

Assuming this default behavior, at this point that a manual intervention is needed. 

Your first priority is to ensure that at least one node is up and serving requests. Once you have accomplished this, you can worry about starting the other nodes, synchronizing them, and so forth. Here is procedure to follow:

1. First, determine which of the scenarios above applies.
2. If the master node A is still responding and serving requests then you have scenario (2) and the problem lies in the connection between master slave. 
   • In this case no emergency measures need to be taken, since the service is still functional, you must simply reestablish the cluster.
   • First, stop the slave B.
   • Ensure that the network problem has been solved.
   • Restart B. It should automatically rejoin the cluster. If it is out-sync you can troubleshoot it according to the tips given in Recovering an Out-Of-Sync Cluster Node.
3. On the other hand, if A is down then you have scenario (1). In this case your first priority shgould be to get functioning instance up and running and serving requests as soon as possible.
   • You have two choices: Restart A and keep it as master or redirect requests to B and make it the new master. Which one you choose should depend on which can be achieved most quickly and with the highest likelyhood of success.
   • If the problem with A is easy to fix then restart it and ensure that it is functioning properly. Then restart B and ensure that it properly rejoins the cluster.
   • If the problem with the master looks more involved, it may be easier to redirect incoming requests to the slave and restart the slave making it he new master.
     • To do this you must first stop B and remove the file
       crx-quickstart/repository/clustered.txt. This is a flag file that CRX creates to keep track of whether a restarted system should regard itself as master or slave. The presence f the file inidicates to the system that before restart it as a slave, so it iwll attempt to automatically rejoin its cluster. The absence of the file indicates that the instance was, before restart, a matsre (or a lone un-clusterd instance), in which case it does not attempt to rejoin any clusters.
     • Now restart B.
     • Once you have confirmed that B is up and running and serving requests you can work on fixing A. When as is in working order you can join it to B, except this time A will be the slave and B the master. Alternatively you may switch back to the original arrangement. Just don't forget abit the clustered.txt file! 
     • It may be possible that A now reports that it is out of sync with cluster node B. See Recovering an Out-Of-Sync Cluster Node for information on fixing this.

As discussed above the default failover behavior in a CRX cluster requires manual intervention because in the typical case a slave cannot distinguish between an inoperable master instance and an inoperable network connection to the master, and it is only the former case that actual failover should be performed.

In some cases, howver, it may make sense to enable automatic failover. This is only recommende if the the slave and master have two or more, distinct but redunadant network connections to one another.

In such a case a slave will only detect heartbeat timeout from the master if either the master is down, or all the redundant network connections are down.

Since the chance of all redunadant conenctions being down at the same time is less than that of a single failed connection, we can have more confidence that a heartbeat timeout really does mean that the matser is down. Obviously this depends on the redundant connections being truly independent, to the greatest extent possible. As well, having even more than two such redundant indepenedent connections would also increase or confidence in the relaiability of the heartbeat monitor.

The level of confidence  in the heartbeat monitor is necessarily a judgement call on the part of the administrator of the cluster. But if it is determined that the level of confidence is high enough, then the default failover behavior can be changed setting to true the parameter becomeMasterOnTimeout in the <Cluster> element of the repository.xml.

Apart from the general guidelines above, there are number of very specific actions to be taken in with respect to particular types of cluster failures. The tables below summarize these scenarios. This specific information should used in concert with procedure above.

When testing a clustered CRX-based project (a website built on CQ for example) the following QA tests should be passed at least once before content entry (with final code but with not all content) and at least once after content entry and before the project goes live:

• Cluster slave shutdown and restart: Verify rejoin to cluster and basic smoke test (sanity check).
• Cluster master shutdown and restart: Verify rejoin to cluster and basic smoke test (sanity check).
• Cluster slave kill -9 or pull plug: Verify rejoin to cluster and basic smoke test (sanity check).
• Cluster master kill -9 or pull plug: Verify rejoin to cluster and basic smoke test (sanity check).
• Recover whole cluster from backup (disaster recovery).