You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mark Miller (Created) (JIRA)" <ji...@apache.org> on 2011/11/07 17:36:51 UTC
[jira] [Created] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Investigate adding an overseer that can assign shards, later do re-balancing, etc
---------------------------------------------------------------------------------
Key: SOLR-2880
URL: https://issues.apache.org/jira/browse/SOLR-2880
Project: Solr
Issue Type: Sub-task
Components: SolrCloud
Reporter: Mark Miller
Assignee: Mark Miller
Fix For: 4.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166679#comment-13166679 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
just got another:
{quote} [junit] Testsuite: org.apache.solr.cloud.OverseerTest
[junit] Testcase: testOverseerFailure(org.apache.solr.cloud.OverseerTest): Caused an ERROR
[junit] KeeperErrorCode = NoNode for /node_states/node1
[junit] org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /node_states/node1
[junit] at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit] at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1228)
[junit] at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:242)
[junit] at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:411)
[junit] at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:316)
[junit] at org.apache.lucene.util.LuceneTestCase$3$1.evaluate(LuceneTestCase.java:528)
[junit] at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:165)
[junit] at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:57)
[junit] Tests run: 3, Failures: 0, Errors: 1, Time elapsed: 5.118 sec {quote}
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165500#comment-13165500 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Got you - my orig plan until we cane up with something better is that you'd always pass the syspeop and up the number when you want to reshard. Not ideal, but worked as a start. I put it in solr.xml because I was handling other props that way but didn't intend for it to stay there. Actually forgot it was there. Currently though, always passing a sys prop seems easier than changing it in zk - of course it doesnt address multiple collections served by the same instance. Long term, none of this seems ideal.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175442#comment-13175442 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Patch applied - thanks again.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
another patch: got rid of CoreAssignment - ZkController now receives shardids through cloudstate, pumped up some counters to get rid of problems with timings, small improvements in OverseerTest
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller resolved SOLR-2880.
-------------------------------
Resolution: Fixed
Thanks Sami!
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165256#comment-13165256 ]
Sami Siren commented on SOLR-2880:
----------------------------------
bq. This will make it easier for people with custom sharding to just add another shard
Yeah, that's a good point. Does the "initial layout creation" code exist anywhere yet?
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165401#comment-13165401 ]
Yonik Seeley commented on SOLR-2880:
------------------------------------
bq. I didn't really like that compared to just passing a sys prop up front
That's what I'm proposing - that numShards not be stored anywhere in ZK at all. Creation of a new collection requires someone creating some initial nodes (i.e. the collection node at a minimum). Currently I believe this is just the bootstrapping code
Scenario 1: autosharding
Whoever creates the new collection also creates a placeholder for each shard. The current bootstrapping code can look at a numShards system property, or a create collection API could have a numShards parameter.
Scenario 2: custom sharding
Create the new collection as normal, but just don't create any placeholders for each shard. Adding a new shard is just
We also shouldn't rely on the number of shards to split up the hash range (except initially when creating a collection) - each shard should advertise what it's range is (in the case of autosharding or sharding by hash). This is important for future splitting of shards (i.e. you could concurrently have shards that covers 0-9, 0-4, and 5-9)
So it seems like numShards is really just an input into new collection creation, not an intrinsic property of the collection.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184183#comment-13184183 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
I wonder if it makes sense for the overseer to change the state of a node in the cluster.json file when a node goes down? It's kind of odd to have it listed as active even when we know a node is out of commission. Should we add another state and start marking nodes as down there?
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165302#comment-13165302 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Now that I got my initial thought down...
After adding your custom shard, you would then have to change the autoShardNum for further replicas to be placed correctly with the current code...
The other option seems to be perhaps starting up all your nodes, each is a new shard, and then you make some call that says you are done adding shards. I didn't really like that compared to just passing a sys prop up front, but perhaps it is better in the end...
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-2880) Investigate adding an
overseer that can assign shards, later do re-balancing, etc
Posted by "Yonik Seeley (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163792#comment-13163792 ]
Yonik Seeley edited comment on SOLR-2880 at 12/6/11 8:15 PM:
-------------------------------------------------------------
For those following along, here's what the ZK layout looks like after starting 4 nodes (with the default 3 shards).
{code}
...../NODE_ASSIGNMENTS (v=0 children=4)
ROGUE:7574_SOLR (v=4) "[{ "_collection":"collection1", "_core":"Rogue:7574_solr_", "shard_name":"shard2"}]"
ROGUE:2222_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:2222_solr_", "shard_name":"shard1"}]"
ROGUE:1111_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:1111_solr_", "shard_name":"shard3"}]"
ROGUE:8983_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:8983_solr_", "shard_name":"shard1"}]"
/NODE_STATES (v=0 children=4)
ROGUE:7574_SOLR (v=4) "[{ "shard_id":"shard2", "_collection":"collection1", "roles":"", "_core":"Rogue:7574_solr_", "s..."
ROGUE:2222_SOLR (v=2) "[{ "shard_id":"shard1", "_collection":"collection1", "roles":"", "_core":"Rogue:2222_solr_", "s..."
ROGUE:1111_SOLR (v=2) "[{ "shard_id":"shard3", "_collection":"collection1", "roles":"", "_core":"Rogue:1111_solr_", "s..."
ROGUE:8983_SOLR (v=2) "[{ "shard_id":"shard1", "_collection":"collection1", "roles":"", "_core":"Rogue:8983_solr_", "s..."
/ZOOKEEPER (v=0 children=1) ""
QUOTA (v=0) ""
/CLUSTERSTATE.JSON (v=10) "{"collection1":{ "shard1":{ "Rogue:2222_solr_":{ "shard_id":"shard1", "_collection":"collection1..."
/LIVE_NODES (v=4 children=3)
ROGUE:2222_SOLR (ephemeral v=0)
ROGUE:1111_SOLR (ephemeral v=0)
ROGUE:8983_SOLR (ephemeral v=0)
/COLLECTIONS (v=1 children=1)
COLLECTION1 (v=0 children=2) "{ "configName":"myconf", "num_shards":"3"}"
SHARDS_LOCK (v=0)
LEADER_ELECT (v=0 children=3)
SHARD1 (v=0 children=2)
ELECTION (v=0 children=2)
N_0000000001 (ephemeral v=0)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:8983_SOLR_ (ephemeral v=0) "{ "shard_id":"shard1", "roles":"", "state":"recovering", "node_name":"Rogue:8983_solr", "url":"..."
SHARD2 (v=0 children=2)
ELECTION (v=0 children=1)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:7574_SOLR_ (ephemeral v=0) "{ "shard_id":"shard2", "roles":"", "state":"recovering", "node_name":"Rogue:7574_solr", "url":"..."
SHARD3 (v=0 children=2)
ELECTION (v=0 children=1)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:1111_SOLR_ (ephemeral v=0) "{ "shard_id":"shard3", "roles":"", "state":"recovering", "node_name":"Rogue:1111_solr", "url":"..."
/OVERSEER_ELECT (v=0 children=2)
ELECTION (v=0 children=4)
N_0000000004 (ephemeral v=0)
N_0000000003 (ephemeral v=0)
N_0000000001 (ephemeral v=0)
N_0000000000 (ephemeral v=0)
LEADER (v=0)
{code}
Notes: Starting up the first node by bootstrapping fails the first time you try. Run it again and it works.
java -Dbootstrap_confdir=./solr/conf -Dcollection.configName=myconf -DzkRun -jar start.jar
was (Author: yseeley@gmail.com):
For those following along, here's what the ZK layout looks like after starting 4 nodes (with the default 3 shards).
{code}
/NODE_ASSIGNMENTS (v=0 children=4)
ROGUE:7574_SOLR (v=4) "[{ "_collection":"collection1", "_core":"Rogue:7574_solr_", "shard_name":"shard2"}]"
ROGUE:2222_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:2222_solr_", "shard_name":"shard1"}]"
ROGUE:1111_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:1111_solr_", "shard_name":"shard3"}]"
ROGUE:8983_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:8983_solr_", "shard_name":"shard1"}]"
/NODE_STATES (v=0 children=4)
ROGUE:7574_SOLR (v=4) "[{ "shard_id":"shard2", "_collection":"collection1", "roles":"", "_core":"Rogue:7574_solr_", "s..."
ROGUE:2222_SOLR (v=2) "[{ "shard_id":"shard1", "_collection":"collection1", "roles":"", "_core":"Rogue:2222_solr_", "s..."
ROGUE:1111_SOLR (v=2) "[{ "shard_id":"shard3", "_collection":"collection1", "roles":"", "_core":"Rogue:1111_solr_", "s..."
ROGUE:8983_SOLR (v=2) "[{ "shard_id":"shard1", "_collection":"collection1", "roles":"", "_core":"Rogue:8983_solr_", "s..."
/ZOOKEEPER (v=0 children=1) ""
QUOTA (v=0) ""
/CLUSTERSTATE.JSON (v=10) "{"collection1":{ "shard1":{ "Rogue:2222_solr_":{ "shard_id":"shard1", "_collection":"collection1..."
/LIVE_NODES (v=4 children=3)
ROGUE:2222_SOLR (ephemeral v=0)
ROGUE:1111_SOLR (ephemeral v=0)
ROGUE:8983_SOLR (ephemeral v=0)
/COLLECTIONS (v=1 children=1)
COLLECTION1 (v=0 children=2) "{ "configName":"myconf", "num_shards":"3"}"
SHARDS_LOCK (v=0)
LEADER_ELECT (v=0 children=3)
SHARD1 (v=0 children=2)
ELECTION (v=0 children=2)
N_0000000001 (ephemeral v=0)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:8983_SOLR_ (ephemeral v=0) "{ "shard_id":"shard1", "roles":"", "state":"recovering", "node_name":"Rogue:8983_solr", "url":"..."
SHARD2 (v=0 children=2)
ELECTION (v=0 children=1)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:7574_SOLR_ (ephemeral v=0) "{ "shard_id":"shard2", "roles":"", "state":"recovering", "node_name":"Rogue:7574_solr", "url":"..."
SHARD3 (v=0 children=2)
ELECTION (v=0 children=1)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:1111_SOLR_ (ephemeral v=0) "{ "shard_id":"shard3", "roles":"", "state":"recovering", "node_name":"Rogue:1111_solr", "url":"..."
/OVERSEER_ELECT (v=0 children=2)
ELECTION (v=0 children=4)
N_0000000004 (ephemeral v=0)
N_0000000003 (ephemeral v=0)
N_0000000001 (ephemeral v=0)
N_0000000000 (ephemeral v=0)
LEADER (v=0)
{code}
Notes: Starting up the first node by bootstrapping fails the first time you try. Run it again and it works.
java -Dbootstrap_confdir=./solr/conf -Dcollection.configName=myconf -DzkRun -jar start.jar
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163792#comment-13163792 ]
Yonik Seeley commented on SOLR-2880:
------------------------------------
For those following along, here's what the ZK layout looks like after starting 4 nodes (with the default 3 shards).
{code}
/NODE_ASSIGNMENTS (v=0 children=4)
ROGUE:7574_SOLR (v=4) "[{ "_collection":"collection1", "_core":"Rogue:7574_solr_", "shard_name":"shard2"}]"
ROGUE:2222_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:2222_solr_", "shard_name":"shard1"}]"
ROGUE:1111_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:1111_solr_", "shard_name":"shard3"}]"
ROGUE:8983_SOLR (v=2) "[{ "_collection":"collection1", "_core":"Rogue:8983_solr_", "shard_name":"shard1"}]"
/NODE_STATES (v=0 children=4)
ROGUE:7574_SOLR (v=4) "[{ "shard_id":"shard2", "_collection":"collection1", "roles":"", "_core":"Rogue:7574_solr_", "s..."
ROGUE:2222_SOLR (v=2) "[{ "shard_id":"shard1", "_collection":"collection1", "roles":"", "_core":"Rogue:2222_solr_", "s..."
ROGUE:1111_SOLR (v=2) "[{ "shard_id":"shard3", "_collection":"collection1", "roles":"", "_core":"Rogue:1111_solr_", "s..."
ROGUE:8983_SOLR (v=2) "[{ "shard_id":"shard1", "_collection":"collection1", "roles":"", "_core":"Rogue:8983_solr_", "s..."
/ZOOKEEPER (v=0 children=1) ""
QUOTA (v=0) ""
/CLUSTERSTATE.JSON (v=10) "{"collection1":{ "shard1":{ "Rogue:2222_solr_":{ "shard_id":"shard1", "_collection":"collection1..."
/LIVE_NODES (v=4 children=3)
ROGUE:2222_SOLR (ephemeral v=0)
ROGUE:1111_SOLR (ephemeral v=0)
ROGUE:8983_SOLR (ephemeral v=0)
/COLLECTIONS (v=1 children=1)
COLLECTION1 (v=0 children=2) "{ "configName":"myconf", "num_shards":"3"}"
SHARDS_LOCK (v=0)
LEADER_ELECT (v=0 children=3)
SHARD1 (v=0 children=2)
ELECTION (v=0 children=2)
N_0000000001 (ephemeral v=0)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:8983_SOLR_ (ephemeral v=0) "{ "shard_id":"shard1", "roles":"", "state":"recovering", "node_name":"Rogue:8983_solr", "url":"..."
SHARD2 (v=0 children=2)
ELECTION (v=0 children=1)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:7574_SOLR_ (ephemeral v=0) "{ "shard_id":"shard2", "roles":"", "state":"recovering", "node_name":"Rogue:7574_solr", "url":"..."
SHARD3 (v=0 children=2)
ELECTION (v=0 children=1)
N_0000000000 (ephemeral v=0)
LEADER (v=0 children=1)
ROGUE:1111_SOLR_ (ephemeral v=0) "{ "shard_id":"shard3", "roles":"", "state":"recovering", "node_name":"Rogue:1111_solr", "url":"..."
/OVERSEER_ELECT (v=0 children=2)
ELECTION (v=0 children=4)
N_0000000004 (ephemeral v=0)
N_0000000003 (ephemeral v=0)
N_0000000001 (ephemeral v=0)
N_0000000000 (ephemeral v=0)
LEADER (v=0)
{code}
Notes: Starting up the first node by bootstrapping fails the first time you try. Run it again and it works.
java -Dbootstrap_confdir=./solr/conf -Dcollection.configName=myconf -DzkRun -jar start.jar
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
Attached patch allows overseer to track shard leaders and update cloud state accordingly. Shard leader is marked with prop leader="true" in its cloudstate props.
Also some of the tests are improved so that they are less likely to fail due to timing issues.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171077#comment-13171077 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Thanks Sami - it's in.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157272#comment-13157272 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Nice, thanks Sami! I'd like to get this committed to the branch quickly so that we don't have to juggle patch updates much.
Do all tests pass yet for you? I see the full distrb test failing - before I look into it, just wanted to make sure it's expected - if so, I'm happy to dig in.
Also, quick comment on the node assignments zookeeper node - do we really need it? Can't we figure out the assignments from the clusterstate info? Does it add anything?
More comments to come.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165416#comment-13165416 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Okay, that's more in line with what I was originally thinking. Nothing actually uses the hsarNim in zk - just something that came in with Samis changes and is floating. In terms of shards advertising ranges, certainly it's something we eventually need to store in zk - it's not necessary until we support adding/removing shards though. It's easy to add when we need it and doesn't give us anything until we do. And you will still need to determine those ranges based on the number of shards.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158403#comment-13158403 ]
Yonik Seeley commented on SOLR-2880:
------------------------------------
bq. However I now see TestRecovery fail constantly when running solr tests:
Probably my fault - I removed the previous version update processor and switched the tests to use the new distributed update processor. I'll try a fresh checkout (too many unrelated changes in my current copy).
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158398#comment-13158398 ]
Sami Siren commented on SOLR-2880:
----------------------------------
bq. I also see some failures that seems unrelated, like: in SoftAutoCommitTest, CoreAdminHandlerTest#testCoreAdminHandler, TestCoreContainer#testPersist
Those went away with svn up. However I now see TestRecovery fail constantly when running solr tests:
{code}
<testcase classname="org.apache.solr.search.TestRecovery" name="testLogReplay" time="60.774">
<failure type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError:
at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:149)
at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:51)
at org.apache.solr.search.TestRecovery.testLogReplay(TestRecovery.java:82)
at org.apache.lucene.util.LuceneTestCase$3$1.evaluate(LuceneTestCase.java:521)
</failure>
</testcase>
{code}
If I only run that single test it passes.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160946#comment-13160946 ]
Sami Siren commented on SOLR-2880:
----------------------------------
bq. Would it be more convenient to store whats on the nodes in node_states and node_assigments as human readable
I agree, I'll change that.
bq. I actually just noticed we use the same ZKNodeProps for the current leader zk node as we use for the node in the cluster state
- not a problem for info that doesn't change, but now that we store the "active,replicating" state there, as well as other properties that could be dynamic, it ends up with stale data as those properties change.
Who uses the information from the leader node (what is actually needed there?)
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166192#comment-13166192 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Committed. Thanks Sami!
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165298#comment-13165298 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
{quote} The number of shards a collection has is always just the number you see in ZK under the collection. This will make it easier for people with custom sharding to just add another shard. Whoever is creating the initial layout should thus create all of the shards at once. {quote}
This seems more complicated to me?
Custom shards are easy at the moment - numShards is only for the auto sharder - other code that needs to know the number of shards (like around hashing) simply counts the number of shards. If you add another shard manually, it should be no problem.
numShards should just be called autoNumShards or something perhaps.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
Here's a patch that fixes a bug that causes some state updates to be missed when overseer node crashes. It also removes the "_" prefixes from the mentioned property names and use numShards as shard count property name.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160500#comment-13160500 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Would it be more convenient to store whats on the nodes in node_states and node_assigments as human readable? Like the ZkNodeProps?
I actually just noticed we use the same ZKNodeProps for the current leader zk node as we use for the node in the cluster state - not a problem for info that doesn't change, but now that we store the "active,replicating" state there, as well as other properties that could be dynamic, it ends up with stale data as those properties change.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184305#comment-13184305 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
I'm seeing OverseerTest fail quite often now:
{noformat}
[junit] Testcase: testShardAssignmentBigger(org.apache.solr.cloud.OverseerTest): FAILED
[junit] could not find counter for shard:null
[junit] junit.framework.AssertionFailedError: could not find counter for shard:null
[junit] at org.apache.solr.cloud.OverseerTest.testShardAssignmentBigger(OverseerTest.java:248)
[junit] at org.apache.lucene.util.LuceneTestCase$3$1.evaluate(LuceneTestCase.java:528)
[junit] at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:165)
[junit] at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:57)
{noformat}
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182754#comment-13182754 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Doh - think i missed this last patch and now it doest apply at all - can you update this to the branch Sami?
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
This patch
-fixes a bug in overseer where when two cores were registered close to each other the edits for the latter would have gone to a stale cloudState.
-the zk nodes the overseer requires are now done in single method call before the live node is created
-some old cruft is also removed
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165337#comment-13165337 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
{quote}Notes: Starting up the first node by bootstrapping fails the first time you try. Run it again and it works.
java -Dbootstrap_confdir=./solr/conf -Dcollection.configName=myconf -DzkRun -jar start.jar {quote}
Whats the failure? I have not seen this.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
the last patch missed some edits, here's a new one.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161833#comment-13161833 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
bq. Who uses the information from the leader node (what is actually needed there?)
just the url right now I think.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161836#comment-13161836 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Why does the overseer class have it's own cloud state and watches on live nodes and stuff? Isn't this all just redundant? The ZkControllers ZkStateReader is already tracking all this stuff and should be the owner of the cloud state, shouldn't it?
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158460#comment-13158460 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Thanks Sami - all tests are passing for me at the moment.
Is there much that is special about the overseer election code vs the leader election code? Seems there is a lot of code dupe there, both in the classes and the tests. Can we merge that up?
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158327#comment-13158327 ]
Sami Siren commented on SOLR-2880:
----------------------------------
bq. Do all tests pass yet for you
I think there might be atleast something timing related: I do not see failures every time. I also see some failures that seems unrelated, like:
in SoftAutoCommitTest, CoreAdminHandlerTest#testCoreAdminHandler, TestCoreContainer#testPersist
b1. Also, quick comment on the node assignments zookeeper node - do we really need it? Can't we figure out the assignments from the clusterstate info? Does it add anything?
The motivation for adding it was that it might reduce the amount of data that needs to be transferred but currently it does not do even that because ZKController still uses the state -> perhaps the assignments node should be dropped.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165429#comment-13165429 ]
Sami Siren commented on SOLR-2880:
----------------------------------
> Nothing actually uses the hsarNim in zk
It's used in shard assignment, if the prop does not exist in collection node the system property is used. I thought it would be nice to have it in configuration so that you'd not need to boot all your nodes with the system property on. But it seems that if there will be a createCollection method somewhere that creates the initial layout the property is not needed.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162811#comment-13162811 ]
Sami Siren commented on SOLR-2880:
----------------------------------
bq. Why does the overseer class have it's own cloud state and watches on live nodes and stuff?
The watch for live nodes is also used for adding watches for node states: when a new node pops up a watch is generated for /node_states/<node-name>
bq. The ZkControllers ZkStateReader is already tracking all this stuff and should be the owner of the cloud state, shouldn't it?
Yeah, makes sense. I'll see how that would work.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166710#comment-13166710 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
I raised the max iterations timeout for the first fail.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159248#comment-13159248 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Committed - thanks Sami!
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183645#comment-13183645 ]
Sami Siren commented on SOLR-2880:
----------------------------------
bq. think i missed this last patch
It seems like it was applied in r1225550.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: (was: current.patch)
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
The correct patch.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164457#comment-13164457 ]
Yonik Seeley commented on SOLR-2880:
------------------------------------
Some random comments so far... Naming:
- numShards vs num_shards... we should try to make system properties consistent with the names that actually appear in ZK
- _core, _collection? why the underscores?
I'm not sure num_shards belongs as a configuration item anywhere (in solr.xml or as a collection property in ZK). The number of shards a collection has is always just the number you see in ZK under the collection. This will make it easier for people with custom sharding to just add another shard. Whoever is creating the initial layout should thus create all of the shards at once.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880.patch
This patch implements a simple overseer (cluster leader). Currently it does not do more than assign shard ids to cores so there is no added functionality. I wanted to post what I currently have here to get some feedback and improvement ideas.
Two new top level zkNodes are created:
/node_assignments
/node_states
Basically the integration point is the ZKController:
When a core is registered its state is registered locally and nodes current state is published under /node_states
When overseer assigns a shard id the state is stored locally (to overseer) and the assignments for the node are published under /node_assignments.
Overseer gets the number of shards required for a collection from the collections properties in ZK. So if you wanted to add a new collection you'd create a new collection node and record the required number of shards (among other things like the used configuration) into its properties.
Even if a node hosts multiple cores it only creates single node (/node_states/<id>) and reads a single node (/node_assignments/<id>). It might be simpler to have multiple subnodes (one for each core)
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Issue Comment Edited] (SOLR-2880) Investigate adding an
overseer that can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165429#comment-13165429 ]
Sami Siren edited comment on SOLR-2880 at 12/8/11 7:08 PM:
-----------------------------------------------------------
> Nothing actually uses the hsarNim in zk
It's used in shard assignment, if the prop does not exist in collection node the system property is used. I thought it would be nice to have it in configuration so that you'd not need to boot the node that runs overseer with the system property on. But it seems that if there will be a createCollection method somewhere that creates the initial layout the property is not needed.
was (Author: siren):
> Nothing actually uses the hsarNim in zk
It's used in shard assignment, if the prop does not exist in collection node the system property is used. I thought it would be nice to have it in configuration so that you'd not need to boot all your nodes with the system property on. But it seems that if there will be a createCollection method somewhere that creates the initial layout the property is not needed.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: current.patch
The attached patch removes the numShards property from the collection node. It also simplifies Overseer by converting it to use ZkStateReader instead of maintaining the read side of cloudstate internally and by removing bunch of not useful code. I also removed the unneeded IOException from method signatures that I added when I converted the serialization to json (they become obsolete when Yonik improved the ser/deser code).
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13165280#comment-13165280 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Thanks Sami! Committed.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Sami Siren (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated SOLR-2880:
-----------------------------
Attachment: SOLR-2880-merge-elections.patch
The attached patch merges the election code. The OverseerElectionTest should be removed because it now has nothing special to test. SliceLeaderElector should perhaps be renamed to LeaderElector or something like that.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166645#comment-13166645 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
just saw this random fail
{quote} [junit] Testcase: testOverseerFailure(org.apache.solr.cloud.OverseerTest): Caused an ERROR
[junit] (null)
[junit] java.lang.NullPointerException
[junit] at org.apache.solr.cloud.OverseerTest.waitForSliceCount(OverseerTest.java:142)
[junit] at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:319)
[junit] at org.apache.lucene.util.LuceneTestCase$3$1.evaluate(LuceneTestCase.java:528)
[junit] at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:165)
[junit] at org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTe{quote}
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183651#comment-13183651 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
Ah, good. I was going to check that - usually I comment after I apply, so I thought I must have missed it.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163809#comment-13163809 ]
Yonik Seeley commented on SOLR-2880:
------------------------------------
Here's the data on some of the bigger nodes:
{code}
/node_assignments/Rogue:7574_solr
[{
"_collection":"collection1",
"_core":"Rogue:7574_solr_",
"shard_name":"shard2"}]
/node_states/Rogue:7574_solr
[{
"shard_id":"shard2",
"_collection":"collection1",
"roles":"",
"_core":"Rogue:7574_solr_",
"state":"active",
"node_name":"Rogue:7574_solr",
"url":"http://Rogue:7574/solr/"}]
/collections/collection1/leader_elect/shard2/leader/Rogue:7574_solr_
{
"shard_id":"shard2",
"roles":"",
"state":"recovering",
"node_name":"Rogue:7574_solr",
"url":"http://Rogue:7574/solr/"}
/clusterstate.json
{"collection1":{
"shard1":{
"Rogue:2222_solr_":{
"shard_id":"shard1",
"_collection":"collection1",
"roles":"",
"_core":"Rogue:2222_solr_",
"state":"active",
"node_name":"Rogue:2222_solr",
"url":"http://Rogue:2222/solr/"},
"Rogue:8983_solr_":{
"shard_id":"shard1",
"_collection":"collection1",
"roles":"",
"_core":"Rogue:8983_solr_",
"state":"active",
"node_name":"Rogue:8983_solr",
"url":"http://Rogue:8983/solr/"}},
"shard2":{"Rogue:7574_solr_":{
"shard_id":"shard2",
"_collection":"collection1",
"roles":"",
"_core":"Rogue:7574_solr_",
"state":"active",
"node_name":"Rogue:7574_solr",
"url":"http://Rogue:7574/solr/"}},
"shard3":{"Rogue:1111_solr_":{
"shard_id":"shard3",
"_collection":"collection1",
"roles":"",
"_core":"Rogue:1111_solr_",
"state":"active",
"node_name":"Rogue:1111_solr",
"url":"http://Rogue:1111/solr/"}}}}
{code}
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2880) Investigate adding an overseer that
can assign shards, later do re-balancing, etc
Posted by "Mark Miller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184559#comment-13184559 ]
Mark Miller commented on SOLR-2880:
-----------------------------------
bq. I'm seeing OverseerTest fail quite often now:
I think I have taken care of this - we now wait longer for a leader to show up when registering compared to the short wait we do for updates.
> Investigate adding an overseer that can assign shards, later do re-balancing, etc
> ---------------------------------------------------------------------------------
>
> Key: SOLR-2880
> URL: https://issues.apache.org/jira/browse/SOLR-2880
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: 4.0
>
> Attachments: SOLR-2880-merge-elections.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch, SOLR-2880.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org