Autoscaling rack awareness

[Richard Goodman <ri...@brandwatch.com>]

Hi, I'm currently running into some trouble trying to set up rack awareness
as a cluster policy.

I run my cluster with 3 way replication, currently a few collection-shards
have 4 replicas, which shows as violations under my current set policies:

{
"set-cluster-policy":[
{
"replica":"<2",
"shard":"#EACH",
"node":"#ANY"
},
{
"replica":0,
"freedisk":"<50",
"strict":false
}
]
}

{
"collection":"collection_name_one",
"shard":"shard12",
"node":"1.2.3.4:8080_solr",
"tagKey":"1.2.3.4:8080_solr",
"violation":{

"replica":"org.apache.solr.client.solrj.cloud.autoscaling.ReplicaCount:{\n
\"NRT\":2,\n  \"PULL\":0,\n  \"TLOG\":0,\n  \"count\":2}",
  "delta":1},
"clause":{
  "replica":"<2",
  "shard":"#EACH",
  "node":"#ANY",
  "collection":"collection_name_one"}
},

I want to implement rack awareness as a policy, there are examples of
availability zone policies, however, not really anything for rack
awareness. Currently we set this when creating a collection:

sysprop.racklocation:*,shard:*,replica:<2

So I tried to implement this via the following policy rule

{"replica": "<2", "shard": "#EACH", "sysprop.racklocation": "*"}

However, this hasn't worked *(because with the extra replication I have
atm, it would certainly raise this as a violation)*, so not sure how I can
implement this?
I saw in the 7.7 docs this following example:
{"replica":"#ALL", "shard":"shard1", "sysprop.rack":"730"}
However, this forces shard 1 of all replicas to belong to a certain rack,
which I don't want to do, I'd rather the replicas have free choice of where
they are placed, providing if two replicas appear on the same racklocation,
it would raise a violation.

Has anyone had experience of setting something like this up, or have any
advice / see an error in my policy set up?

*(Currently running solr 7.4)*

Thanks,
Richard

Re: Autoscaling rack awareness

[Richard Goodman <ri...@brandwatch.com>]

So I managed to get this working by the following policy:

{"replica":"<2","shard":"#EACH","sysprop.racklocation": "#EACH"}


On Tue, 26 Mar 2019 at 14:03, Richard Goodman <ri...@brandwatch.com>
wrote:

> Hi, I'm currently running into some trouble trying to set up rack
> awareness as a cluster policy.
>
> I run my cluster with 3 way replication, currently a few collection-shards
> have 4 replicas, which shows as violations under my current set policies:
>
> {
> "set-cluster-policy":[
> {
> "replica":"<2",
> "shard":"#EACH",
> "node":"#ANY"
> },
> {
> "replica":0,
> "freedisk":"<50",
> "strict":false
> }
> ]
> }
>
> {
> "collection":"collection_name_one",
> "shard":"shard12",
> "node":"1.2.3.4:8080_solr",
> "tagKey":"1.2.3.4:8080_solr",
> "violation":{
>
> "replica":"org.apache.solr.client.solrj.cloud.autoscaling.ReplicaCount:{\n
> \"NRT\":2,\n  \"PULL\":0,\n  \"TLOG\":0,\n  \"count\":2}",
>   "delta":1},
> "clause":{
>   "replica":"<2",
>   "shard":"#EACH",
>   "node":"#ANY",
>   "collection":"collection_name_one"}
> },
>
> I want to implement rack awareness as a policy, there are examples of
> availability zone policies, however, not really anything for rack
> awareness. Currently we set this when creating a collection:
>
> sysprop.racklocation:*,shard:*,replica:<2
>
> So I tried to implement this via the following policy rule
>
> {"replica": "<2", "shard": "#EACH", "sysprop.racklocation": "*"}
>
> However, this hasn't worked *(because with the extra replication I have
> atm, it would certainly raise this as a violation)*, so not sure how I
> can implement this?
> I saw in the 7.7 docs this following example:
> {"replica":"#ALL", "shard":"shard1", "sysprop.rack":"730"}
> However, this forces shard 1 of all replicas to belong to a certain rack,
> which I don't want to do, I'd rather the replicas have free choice of where
> they are placed, providing if two replicas appear on the same racklocation,
> it would raise a violation.
>
> Has anyone had experience of setting something like this up, or have any
> advice / see an error in my policy set up?
>
> *(Currently running solr 7.4)*
>
> Thanks,
> Richard
>


-- 

Richard Goodman    |    Data Infrastructure Engineer

richardg@brandwatch.com


NEW YORK   | BOSTON  | BRIGHTON   | LONDON   | BERLIN   |   STUTTGART   |
SINGAPORE   | SYDNEY | PARIS


<https://www.brandwatch.com/blog/brandwatch-and-crimson-hexagon/>