[jira] [Updated] (SOLR-12993) Split the state.json into 2. a small frequently modified data + a large unmodified data

["Noble Paul (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Noble Paul updated SOLR-12993:
------------------------------
    Description: 
The new design is posted here
https://docs.google.com/document/d/1AZyiRA_bRhAWkUM1Nj5kg__xpPM9Fd_iwHhYazG38xI/edit#

  was:
This a just a proposal to minimize the ZK load and improve scalability of very large clusters.

Every time a small state change occurs for a collection/replica the following file needs to be updated + read * n times (where n = no of replicas for this collection ). The proposal is to split the main file into 2.
{code}
{"gettingstarted":{
    "pullReplicas":"0",
    "replicationFactor":"2",
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"-1",
    "autoAddReplicas":"false",
    "nrtReplicas":"2",
    "tlogReplicas":"0",
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
      
        "replicas":{
          "core_node3":{
            "core":"gettingstarted_shard1_replica_n1",
            "base_url":"http://10.0.0.80:8983/solr",
            "node_name":"10.0.0.80:8983_solr",
            "state":"active",
            "type":"NRT",
            "force_set_state":"false",
            "leader":"true"},
          "core_node5":{
            "core":"gettingstarted_shard1_replica_n2",
            "base_url":"http://10.0.0.80:7574/solr",
            "node_name":"10.0.0.80:7574_solr",
         
            "type":"NRT",
            "force_set_state":"false"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node7":{
            "core":"gettingstarted_shard2_replica_n4",
            "base_url":"http://10.0.0.80:7574/solr",
            "node_name":"10.0.0.80:7574_solr",
           
            "type":"NRT",
            "force_set_state":"false"},
          "core_node8":{
            "core":"gettingstarted_shard2_replica_n6",
            "base_url":"http://10.0.0.80:8983/solr",
            "node_name":"10.0.0.80:8983_solr",
         
            "type":"NRT",
            "force_set_state":"false",
            "leader":"true"}}}}}}
{code}
another file {{status.json}} which is frequently updated and small.
{code}
{
    "shard1": {
      "state": "ACTIVE",
      "core_node3": {"state": "active", "leader" : true},
      "core_node5": {"state": "active"}
    },
    "shard2": {
      "state": "active",
      "core_node7": {"state": "active"},
      "core_node8": {"state": "active", "leader" : true}}
  }
{code}
Here the size of the file is roughly one tenth of the other file. This leads to a dramatic reduction in the amount of data written/read to/from ZK.


> Split the state.json into 2. a small frequently modified data + a large unmodified data
> ---------------------------------------------------------------------------------------
>
>                 Key: SOLR-12993
>                 URL: https://issues.apache.org/jira/browse/SOLR-12993
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Noble Paul
>            Priority: Major
>
> The new design is posted here
> https://docs.google.com/document/d/1AZyiRA_bRhAWkUM1Nj5kg__xpPM9Fd_iwHhYazG38xI/edit#



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org