You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Noble Paul (Jira)" <ji...@apache.org> on 2019/10/28 20:13:00 UTC
[jira] [Commented] (SOLR-12993) Split the state.json into 2. a
small frequently modified data + a large unmodified data
[ https://issues.apache.org/jira/browse/SOLR-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961435#comment-16961435 ]
Noble Paul commented on SOLR-12993:
-----------------------------------
I plan to explore this option now. The new design is added to the [doc|[https://docs.google.com/document/d/1AZyiRA_bRhAWkUM1Nj5kg__xpPM9Fd_iwHhYazG38xI/edit#]]
> Split the state.json into 2. a small frequently modified data + a large unmodified data
> ---------------------------------------------------------------------------------------
>
> Key: SOLR-12993
> URL: https://issues.apache.org/jira/browse/SOLR-12993
> Project: Solr
> Issue Type: Improvement
> Reporter: Noble Paul
> Priority: Major
>
> This a just a proposal to minimize the ZK load and improve scalability of very large clusters.
> Every time a small state change occurs for a collection/replica the following file needs to be updated + read * n times (where n = no of replicas for this collection ). The proposal is to split the main file into 2.
> {code}
> {"gettingstarted":{
> "pullReplicas":"0",
> "replicationFactor":"2",
> "router":{"name":"compositeId"},
> "maxShardsPerNode":"-1",
> "autoAddReplicas":"false",
> "nrtReplicas":"2",
> "tlogReplicas":"0",
> "shards":{
> "shard1":{
> "range":"80000000-ffffffff",
>
> "replicas":{
> "core_node3":{
> "core":"gettingstarted_shard1_replica_n1",
> "base_url":"http://10.0.0.80:8983/solr",
> "node_name":"10.0.0.80:8983_solr",
> "state":"active",
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"},
> "core_node5":{
> "core":"gettingstarted_shard1_replica_n2",
> "base_url":"http://10.0.0.80:7574/solr",
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"}}},
> "shard2":{
> "range":"0-7fffffff",
> "state":"active",
> "replicas":{
> "core_node7":{
> "core":"gettingstarted_shard2_replica_n4",
> "base_url":"http://10.0.0.80:7574/solr",
> "node_name":"10.0.0.80:7574_solr",
>
> "type":"NRT",
> "force_set_state":"false"},
> "core_node8":{
> "core":"gettingstarted_shard2_replica_n6",
> "base_url":"http://10.0.0.80:8983/solr",
> "node_name":"10.0.0.80:8983_solr",
>
> "type":"NRT",
> "force_set_state":"false",
> "leader":"true"}}}}}}
> {code}
> another file {{status.json}} which is frequently updated and small.
> {code}
> {
> "shard1": {
> "state": "ACTIVE",
> "core_node3": {"state": "active", "leader" : true},
> "core_node5": {"state": "active"}
> },
> "shard2": {
> "state": "active",
> "core_node7": {"state": "active"},
> "core_node8": {"state": "active", "leader" : true}}
> }
> {code}
> Here the size of the file is roughly one tenth of the other file. This leads to a dramatic reduction in the amount of data written/read to/from ZK.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org