Script for requesting recovery on down replicas

[Walter Underwood <wu...@wunderwood.org>]

We have a high update rate collection with a lot of replicas. Sometimes after a config reload, some of the replicas go down (brown in the cloud graph). I got really tired of fixing the by hand in a 40 node cluster.

I wrote a script to mine those out of clusterstatus and send a request recovery command for each one. You’ll need “jq” to run this. I’m putting it in the body because attachments are stripped on this list. I named it “request-recovery.sh”. The hardest part of this was dealing with arrays in bash.

=========
#!/bin/bash

cluster=$1

if [ -z "$cluster" ]
then
    echo "Must provide a hostname from Solr Cloud cluster as the first argument"
    echo "usage: ./request-recovery.sh solr-cloud.mydomain.com collection_name"
    exit 1
fi

collection=$2

if [ -z "$collection" ]
then
    echo "Must provide a Solr collection name as the second argument"
    echo "usage: ./request-recovery.sh solr-cloud.mydomain.com collection_name"
    exit 1
fi

# Fetch the hostnames (node_names) and core names of the cores in $collection
# which are in the "down" state. Store those in two parallel arrays.
# We create arrays by wrapping the curl calls in ().

down_node_names=(`curl -s "http://${cluster}:8983/solr/admin/collections?action=CLUSTERSTATUS&wt=json" | jq -r ".cluster.collections.$collection.shards[].replicas[] | select(.state==\"down\") | .node_name"`)
down_cores=(`curl -s "http://${cluster}:8983/solr/admin/collections?action=CLUSTERSTATUS&wt=json" | jq -r ".cluster.collections.$collection.shards[].replicas[] | select(.state==\"down\") | .core"`)

echo "${#down_node_names[@]} cores are down in collection $collection"

# ${!array[@]} is the list of all the indexes set in the array
for i in ${!down_node_names[@]}
do
    echo "Requesting recovery for core ${down_cores[i]} on node ${down_node_names[i]}"
    url_frag=`echo "${down_node_names[i]}" | tr _ /` 
    curl "http://$url_frag/admin/cores?action=REQUESTRECOVERY&core=${down_cores[i]}&wt=json"
done
=========

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)