Index Replication distributes complete copies of a master index to one or more slave servers. The master server continues to manage updates to the index. All querying is handled by the slaves. This division of labor enables Solr to scale to provide adequate responsiveness to queries against large search volumes.
The figure below shows a Solr configuration using index replication. The master server’s index is replicated on the slaves.
Index Replication in Solr
Solr includes a Java implementation of index replication that works over HTTP.
For information on the ssh/rsync based replication, see Index Replication using ssh and rsync.
The Java-based implementation of index replication offers these benefits:
- Replication without requiring external scripts
- The configuration affecting replication is controlled by a single file, solrconfig.xml
- Supports the replication of configuration files as well as index files
- Works across platforms with same configuration
- No reliance on OS-dependent hard links
- Tightly integrated with Solr; an admin page offers fine-grained control of each aspect of replication
- The Java-based replication feature is implemented as a RequestHandler. Configuring replication is therefore similar to any normal RequestHandler.
The table below defines the key terms associated with Solr replication.
|Collection||A Lucene collection is a directory of files. These files make up the indexed and returnable data of a Solr search repository.|
|Distribution||The copying of a collection from the master server to all slaves. The distribution process takes advantage of Lucene’s index file structure.|
|Inserts and Deletes||As inserts and deletes occur in the collection, the directory remains unchanged. Documents are always inserted into newly created files. Documents that are deleted are not removed from the files. They are flagged in the file, deletable, and are not removed from the files until the collection is optimized.|
|Master and Slave||The Solr distribution model uses the master/slave model. The master is the service which receives all updates initially and keeps everything organized. Solr uses a single update master server coupled with multiple query slave servers. All changes (such as inserts, updates, deletes, etc.) are made against the single master server. Changes made on the master are distributed to all the slave servers which service all query requests from the clients.|
|Update||An update is a single change request against a single Solr instance. It may be a request to delete a document, add a new document, change a document, delete all documents matching a query, etc. Updates are handled synchronously within an individual Solr instance.|
|Optimization||A process that compacts the index and merges segments in order to improve query performance. New secondary segment(s) are created to contain documents inserted into the collection after it has been optimized. A Lucene collection must be optimized periodically to maintain satisfactory query performance. Optimization is run on the master server only. An optimized index will give you a performance gain at query time of at least 10%. This gain may be more on an index that has become fragmented over a period of time with many updates and no optimizations. Optimizations require a much longer time than does the distribution of an optimized collection to all slaves.|
|Segments||The number of files in a collection.|
|mergeFactor||A parameter that controls the number of files (segments) in a collection. For example, when mergeFactor is set to 3, Solr will fill one segment with documents until the limit maxBufferedDocs is met, then it will start a new segment. When the number of segments specified by mergeFactor is reached–|
|Snapshot||A directory containing hard links to the data files. Snapshots are distributed from the master server when the slaves pull them, “smartcopying” the snapshot directory that contains the hard links to the most recent collection data files.|
Configuring the Replication RequestHandler on a Master Server
Before running a replication, you should set the following parameters on initialization of the handler:
|replicateAfter||String specifying action after which replication should occur. Valid values are commit, optimize, or startup. There can be multiple values for this parameter. If you use “startup”, you need to have a “commit” and/or “optimize” entry also if you want to trigger replication on future commits or optimizes.|
|backupAfter||String specifying action after which a backup should occur. Valid values are commit, optimize, or startup. There can be multiple values for this parameter. It is not required for replication, it just makes a backup.|
|maxNumberOfBackups||Integer specifying how many backups to keep. This can be used to delete all but the most recent N backups.|
|confFiles||The configuration files to replicate, separated by a comma.|
|commitReserveDuration||If your commits are very frequent and your network is slow, you can tweak this parameter to increase the amount of time taken to download 5Mb from the master to a slave. The default is 10 seconds.|
The example below shows how to configure the Replication RequestHandler on a master server.
In the configuration file on the master server, include a line like the following:
This ensures that the local configuration solrconfig_slave.xml will be saved as solrconfig.xml on the slave. All other files will be saved with their original names.
On the master server, the file name of the slave configuration file can be anything, as long as the name is correctly identified in the confFiles string; then it will be saved as whatever file name appears after the colon ‘:’.
Configuring the Replication RequestHandler on a Slave Server
The code below shows how to configure a ReplicationHandler on a slave.
|If you are not using cores, then you simply omit the corename parameter above in the masterUrl. To ensure that the URL is correct, just hit the URL with a browser. You must get a status OK response.|
A master may be able to serve only so many slaves without affecting performance. Some organizations have deployed slave servers across multiple data centers. If each slave downloads the index from a remote data center, the resulting download may consume too much network bandwidth. To avoid performance degradation in cases like this, you can configure one or more slaves as repeaters. A repeater is simply a node that acts as both a master and a slave.
- To configure a server as a repeater, the definition of the Replication requestHandler in the solrconfig.xml file must include file lists of use for both masters and slaves.
- Be sure to set the replicateAfter parameter to commit, even if replicateAfter is set to optimize on the main master. This is because on a repeater (or any slave), a commit is called only after the index is downloaded. The optimize command is never called on slaves.
- Optionally, one can configure the repeater to fetch compressed files from the master through the compression parameter to reduce the index download time.
Here is an example of a ReplicationHandler configuration for a repeater:
Commit and Optimize Operations
When a commit or optimize operation is performed on the master, the RequestHandler reads the list of file names which are associated with each commit point. This relies on the replicateAfter parameter in the configuration to decide which types of events should trigger replication.
|Setting on the Master||Description|
|commit||Triggers replication whenever a commit is performed on the master index.|
|optimize||Triggers replication whenever the master index is optimized.|
|startup||Triggers replication whenever the master index starts up.|
The replicateAfter parameter can accept multiple arguments. For example:
The master is totally unaware of the slaves. The slave continuously keeps polling the master (depending on the pollInterval parameter) to check the current index version of the master. If the slave finds out that the master has a newer version of the index it initiates a replication process. The steps are as follows:
- The slave issues a filelist command to get the list of the files. This command returns the names of the files as well as some metadata (for example, size, a lastmodified timestamp, an alias if any).
- The slave checks with its own index if it has any of those files in the local index. It then runs the filecontent command to download the missing files. This uses a custom format (akin to the HTTP chunked encoding) to download the full content or a part of each file. If the connection breaks in between , the download resumes from the point it failed. At any point, the slave tries 5 times before giving up a replication altogether.
- The files are downloaded into a temp directory, so that if either the slave or the master crashes during the download process, no files will be corrupted. Instead, the current replication will simply abort.
- After the download completes, all the new files are moved to the live index directory and the file’s timestamp is same as its counterpart on the master.
- A commit command is issued on the slave by the Slave’s ReplicationHandler and the new index is loaded.
Replicating Configuration Files
To replicate configuration files, list them using using the confFiles parameter. Only files found in the conf directory of the master’s Solr instance will be replicated.
Solr replicates configuration files only when the index itself is replicated. That means even if a configuration file is changed on the master, that file will be replicated only after there is a new commit/optimize on master’s index.
Unlike the index files, where the timestamp is good enough to figure out if they are identical, configuration files are compared against their checksum. The schema.xml files (on master and slave) are judged to be identical if their checksums are identical.
As a precaution when replicating configuration files, Solr copies configuration files to a temporary directory before moving them into their ultimate location in the conf directory. The old configuration files are then renamed and kept in the same conf/ directory. The ReplicationHandler does not automatically clean up these old files.
If a replication involved downloading of at least one configuration file, the ReplicationHandler issues a core-reload command instead of a commit command.
Resolving Corruption Issues on Slave Servers
If documents are added to the slave, then the slave is no longer in sync with its master. However, the slave will not undertake any action to put itself in sync, until the master has new index data. When a commit operation takes place on the master, the index version of the master becomes different from that of the slave. The slave then fetches the list of files and finds that some of the files present on the master are also present in the local index but with different sizes and timestamps. This means that the master and slave have incompatible indexes. To correct this problem, the slave then copies all the index files from master to a new index directory and asks the core to load the fresh index from the new directory.
HTTP API Commands for the ReplicationHandler
You can use the HTTP commands below to control the ReplicationHandler’s operations.
|http://_master_host_:_port_/solr/replication?command=enablereplication||Enables replication on the master for all its slaves.|
|http://_master_host_:_port_/solr/replication?command=disablereplication||Disables replication on the master for all its slaves.|
|http://_host_:_port_/solr/replication?command=indexversion||Returns the version of the latest replicatable index on the specified master or slave.|
|http://_slave_host_:_port_/solr/replication?command=fetchindex||Forces the specified slave to fetch a copy of the index from its master. |
If you like, you can pass an extra attribute such as masterUrl or compression (or any other parameter which is specified in the <lst name="slave"> tag) to do a one time replication from a master. This obviates the need for hard-coding the master in the slave.
|http://_slave_host_:_port_/solr/replication?command=abortfetch||Aborts copying an index from a master to the specified slave.|
|http://_slave_host_:_port_/solr/replication?command=enablepoll||Enables the specified slave to poll for changes on the master.|
|http://_slave_host_:_port_/solr/replication?command=disablepoll||Disables the specified slave from polling for changes on the master.|
|http://_slave_host_:_port_/solr/replication?command=details||Retrieves configuration details and current status.|
|http://host:port/solr/replication?command=filelist&indexversion=<index-version-number>||Retrieves a list of Lucene files present in the specified host’s index. You can discover the version number of the index by running the indexversion command.|
|http://_master_host_:_port_/solr/replication?command=backup||Creates a backup on master if there are committed index data in the server; otherwise, does nothing. This command is useful for making periodic backups. The numberToKeep request parameter can be used with the backup command unless the maxNumberOfBackups initialization parameter has been specified on the handler – in which case maxNumberOfBackups is always used and attempts to use the numberToKeep request parameter will cause an error.|
Index Replication using ssh and rsync
Solr supports ssh/rsync-based replication. This mechanism only works on systems that support removing open hard links.
Solr distribution is similar in concept to database replication. All collection changes come to one master Solr server. All production queries are done against query slaves. Query slaves receive all their collection changes indirectly — as new versions of a collection which they pull from the master. These collection downloads are polled for on a cron’d basis.
A collection is a directory of many files. Collections are distributed to the slaves as snapshots of these files. Each snapshot is made up of hard links to the files so copying of the actual files is not necessary when snapshots are created. Lucene only significantly rewrites files following an optimization command. Generally, once a file is written, it will change very little, if at all. This makes the underlying transport of rsync very useful. Files that have already been transferred and have not changed do not need to be re-transferred with the new edition of a collection.
The Snapshot and Distribution Process
Here are the steps that Solr follows when replicating an index:
- The snapshooter command takes snapshots of the collection on the master. It runs when invoked by Solr after it has done a commit or an optimize.
- The snappuller command runs on the query slaves to pull the newest snapshot from the master. This is done via rsync in daemon mode running on the master for better performance and lower CPU utilization over rsync using a remote shell program as the transport.
- The snapinstaller runs on the slave after a snapshot has been pulled from the master. This signals the local Solr server to open a new index reader, then auto-warming of the cache(s) begins (in the new reader), while other requests continue to be served by the original index reader. Once auto-warming is complete, Solr retires the old reader and directs all new queries to the newly cache-warmed reader.
- All distribution activity is logged and written back to the master to be viewable on the distribution page of its GUI.
- Old versions of the index are removed from the master and slave servers by a cron’d snapcleaner.
If you are building an index from scratch, distribution is the final step of the process.
Manual copying of index files is not recommended; however, running distribution commands manually (that is, not relying on crond to run them) is perfectly fine.
Snapshots are stored in directories whose names follow this format: snapshot.yyyymmddHHMMSS
All the files in the index directory are hard links to the latest snapshot. This design offers these advantages:
- The Solr implementation can keep multiple snapshots on each host without needing to keep multiple copies of index files that have not changed.
- File copying from master to slave is very fast.
- Taking a snapshot is very fast as well.
Solr Distribution Scripts
For the Solr distribution scripts, the name of the index directory is defined either by the environment variable data_dir in the configuration file solr/conf/scripts.conf or the command line argument -d. It should match the value used by the Solr server which is defined in solr/conf/solrconfig.xml.
All Solr collection distribution scripts are bundled in a Solr release and reside in the directory solr/src/scripts. It’s recommended that you install the scripts in a solr/bin/ directory.
Collection distribution scripts create and prepare for distribution a snapshot of a search collection after each commit and optimize request if the postCommit and postOptimize event listener is configured in solrconfig.xml to execute snapshooter.
The snapshooter script creates a directory snapshot.<ts>, where <ts> is a timestamp in the format, yyyymmddHHMMSS. It contains hard links to the data files.
Snapshots are distributed from the master server when the slaves pull them, “smartcopying” the snapshot directory that contains the hard links to the most recent collection data files.
|snapshooter||Creates a snapshot of a collection. Snapshooter is normally configured to run on the master Solr server when a commit or optimize happens. Snapshooter can also be run manually, but one must make sure that the index is in a consistent state, which can only be done by pausing indexing and issuing a commit.|
|snappuller||A shell script that runs as a cron job on a slave Solr server. The script looks for new snapshots on the master Solr server and pulls them.|
|snappuller-enable||Creates the file solr/logs/snappuller-enabled, whose presence enables snappuller.|
|snapinstaller||Installs the latest snapshot (determined by the timestamp) into the place, using hard links (similar to the process of taking a snapshot). Then solr/logs/snapshot.current is written and scp’d (secure copied) back to the master Solr server. snapinstaller then triggers the Solr server to open a new Searcher.|
|snapcleaner||Runs as a cron job to remove snapshots more than a configurable number of days old or all snapshots except for the most recent n number of snapshots. Also can be run manually.|
|rsyncd-start||Starts the rsyncd daemon on the master Solr server which handles collection distribution requests from the slaves.|
|rsyncd daemon||Efficiently synchronizes a collection–|
|rsyncd-stop||Stops the rsyncd daemon on the master Solr server. The stop script then makes sure that the daemon has in fact exited by trying to connect to it for up to 300 seconds. The stop script exits with error code 2 if it fails to stop the rsyncd daemon.|
|rsyncd-enable||Creates the file solr/logs/rsyncd-enabled, whose presence allows the rsyncd daemon to run, allowing replication to occur.|
|rsyncd-disable||Removes the file solr/logs/rsyncd-enabled, whose absence prevents the rsyncd daemon from running, preventing replication.|
For more information about usage arguments and syntax see the SolrCollectionDistributionScripts page on the Solr Wiki.
Solr Distribution-related Cron Jobs
The distribution process is automated through the use of cron jobs. The cron jobs should run under the user ID that the Solr server is running under.
|snapcleaner||The snapcleaner job should be run out of cron at the regular basis to clean up old snapshots. This should be done on both the master and slave Solr servers. For example, the following cron job runs everyday at midnight and cleans up snapshots 8 days and older: |
0 0 * * * <solr.solr.home>/solr/bin/snapcleaner -D 7
Additional cleanup can always be performed on-demand by running snapcleaner manually.
|snappuller snapinstaller||On the slave Solr servers, snappuller should be run out of cron regularly to get the latest index from the master Solr server. It is a good idea to also run snapinstaller with snappuller back-to-back in the same crontab entry to install the latest index once it has been copied over to the slave Solr server.|
For example, the following cron job runs every 5 minutes to keep the slave Solr server in sync with the master Solr server:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * <solr.solr.home>/solr/bin/snappuller;<solr.solr.home>/solr/bin/snapinstaller
|Modern cron allows this to be shortened to */5 * * * *....|
Performance Tuning for Script-based Replication
Because fetching a master index uses the rsync utility, which transfers only the segments that have changed, replication is normally very fast. However, if the master server has been optimized, then rsync may take a long time, because many segments will have been changed in the process of optimization.
- If replicating to multiple slaves consumes too much network bandwidth, consider the use of a repeater.
- Make sure that slaves do not pull from the master so frequently that a previous replication is still running when a new one is started. In general, it’s best to allow at least a minute for the replication process to complete. But in configurations with low network bandwidth or a very large index, even more time may be required.
Commit and Optimization
On a very large index, adding even a few documents and then running an optimize operation causes the complete index to be rewritten. This consumes a lot of disk I/O and impacts query performance. Optimizing a very large index may even involve copying the index twice and calling optimize at the beginning and at the end. If some documents have been deleted, the first optimize call will rewrite the index even before the second index is merged.
Optimization is an I/O intensive process, as the entire index is read and re-written in optimized form. Anecdotal data shows that optimizations on modest server hardware can take around 5 minutes per GB, although this obviously varies considerably with index fragmentation and hardware bottlenecks. We do not know what happens to query performance on a collection that has not been optimized for a long time. We do know that it will get worse as the collection becomes more fragmented, but how much worse is very dependent on the manner of updates and commits to the collection. The setting of the mergeFactor attribute affects performance as well. Dividing a large index with millions of documents into even as few as five segments may degrade search performance by as much as 15-20%.
While optimizing has many benefits, a rapidly changing index will not retain those benefits for long, and since optimization is an intensive process, it may be better to consider other options, such as lowering the merge factor (discussed in this Guide in the section on Configuring the Lucene Index Writers).
Distribution and Optimization
The time required to optimize a master index can vary dramatically. A small index may be optimized in minutes. A very large index may take hours. The variables include the size of the index and the speed of the hardware.
Distributing a newly optimized collection may take only a few minutes or up to an hour or more, again depending on the size of the index and the performance capabilities of network connections and disks. During optimization the machine is under load and does not process queries very well. Given a schedule of updates being driven a few times an hour to the slaves, we cannot run an optimize with every committed snapshot.
Copying an optimized collection means that the entire collection will need to be transferred during the next snappull. This is a large expense, but not nearly as huge as running the optimize everywhere. Consider this example: on a three-slave one-master configuration, distributing a newly-optimized collection takes approximately 80 seconds total. Rolling the change across a tier would require approximately ten minutes per machine (or machine group). If this optimize were rolled across the query tier, and if each collection being optimized were disabled and not receiving queries, a rollout would take at least twenty minutes and potentially as long as an hour and a half. Additionally, the files would need to be synchronized so that the following rsync, snappull would not think that the independently optimized files were different in any way. This would also leave the door open to independent corruption of collections instead of each being a perfect copy of the master.
Optimizing on the master allows for a straight-forward optimization operation. No query slaves need to be taken out of service. The optimized collection can be distributed in the background as queries are being normally serviced. The optimization can occur at any time convenient to the application providing collection updates.