You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov> on 2015/11/18 20:20:26 UTC
Implementing security.json is breaking ADDREPLICA
Implementing security.json is breaking ADDREPLICA
I have been able to reproduce this issue with minimal changes from an out-of-the-box Zookeeper (3.4.6) and Solr (5.3.1): loading configsets/basic_configs/conf into Zookeeper, creating the security.json listed below, creating two nodes (one with a core named xmpl and one without any core)- I can provide details if helpful.
The security.json is as follows:
{
"authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":{
"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
"solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE= 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
"":{"v":9}},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{
"solr":[
"admin",
"read",
"xmpladmin",
"xmplgen",
"xmplsel"],
"solruser":[
"read",
"xmplgen",
"xmplsel"]},
"permissions":[
{
"name":"security-edit",
"role":"admin"},
{
"name":"xmpl_admin",
"collection":"xmpl",
"path":"/admin/*",
"role":"xmpladmin"},
{
"name":"xmpl_sel",
"collection":"xmpl",
"path":"/select/*",
"role":null},
{
"name":"xmpl_gen",
"collection":"xmpl",
"path":"/*",
"role":"xmplgen"}],
"":{"v":42}}}
When I then execute admin/collections?action=ADDREPLICA, I get errors such as the following in the solr.log of the node which was created without a core.
INFO - 2015-11-17 21:03:54.157; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery.
INFO - 2015-11-17 21:03:54.158; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates.
INFO - 2015-11-17 21:03:54.158; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
INFO - 2015-11-17 21:03:54.159; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://{IP-address-redacted}:4565/solr/xmpl/.
ERROR - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.common.SolrException; Error while trying to recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://{IP-address-redacted}:4565/solr/xmpl: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Unauthorized request, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/xmpl/update. Reason:
<pre> Unauthorized request, Response code: 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
INFO - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.update.UpdateLog; Dropping buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
ERROR - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (2)
INFO - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2 x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0 seconds before trying to recover again (3)
And (after modifying Logging Levels), the solr.log of the node which already had a core gets errors such as the following:
2015-11-17 21:03:50.743 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server REQUEST GET /solr/tpl/cloud.html on HttpChannelOverHttp@37cf94f4{r=1,c=false,a=DISPATCHED,uri=/solr/tpl/cloud.html}
2015-11-17 21:03:50.744 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server RESPONSE /solr/tpl/cloud.html 200 handled=true
2015-11-17 21:03:50.802 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4{r=2,c=false,a=DISPATCHED,uri=/solr/zookeeper}
2015-11-17 21:03:50.803 INFO (qtp59559151-91) [ ] o.a.s.s.HttpSolrCall userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
2015-11-17 21:03:50.831 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server RESPONSE /solr/zookeeper 200 handled=true
2015-11-17 21:03:50.837 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4{r=3,c=false,a=DISPATCHED,uri=/solr/zookeeper}
2015-11-17 21:03:50.838 INFO (qtp59559151-87) [ ] o.a.s.s.HttpSolrCall userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
2015-11-17 21:03:50.841 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server RESPONSE /solr/zookeeper 200 handled=true
2015-11-17 21:03:50.857 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4{r=4,c=false,a=DISPATCHED,uri=/solr/zookeeper}
2015-11-17 21:03:50.858 INFO (qtp59559151-91) [ ] o.a.s.s.HttpSolrCall userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
2015-11-17 21:03:50.860 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server RESPONSE /solr/zookeeper 200 handled=true
2015-11-17 21:03:54.162 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server REQUEST POST /solr/xmpl/update on HttpChannelOverHttp@1cf967f0{r=1,c=false,a=DISPATCHED,uri=/solr/xmpl/update}
2015-11-17 21:03:54.164 INFO (qtp59559151-87) [c:xmpl s:shard1 r:core_node1 x:xmpl] o.a.s.s.HttpSolrCall userPrincipal: [null] type: [WRITE], collections: [xmpl,], Path: [/update]
2015-11-17 21:03:54.164 DEBUG (qtp59559151-87) [c:xmpl s:shard1 r:core_node1 x:xmpl] o.e.j.s.Server RESPONSE /solr/xmpl/update 401 handled=true
My impression from Anshum Gupta's 10/16/15 talk in Austin at the Solr conference was that this was supposed to work. It does seem that one might be able to add security to replication, but there does not seem to be a way to add SolrCloud replication to this type of security.
Also, on a side note, I notice that http://hostname:port/solr/ does bring up the GUI without prompting for a password: the Security team here would like us to implement security.json in such a way that even the front page of the GUI will require a password (although they will allow us to allow select access without a password): I have not yet found a way via security.json to implement that a password would be required in order to access the GUI front page.
Please advise.
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
>From my tests, it seems like the 'read' permission interferes with the
Replication and so the ADDREPLICA also fails. You're also bound to run into
issues if you have 'read' permission setup and restart your cluster,
provided you have a collection that has a replication factor > 1 for at
least one shard.
I'll create a JIRA for this and mark it to be a blocker for 5.4. Thanks for
bringing this up.
On Thu, Nov 19, 2015 at 12:43 PM, Anshum Gupta <an...@anshumgupta.net>
wrote:
> I'll try out what you did later in the day, as soon as I get time but why
> exactly are you creating cores manually? Seems like you manually create a
> core and the try to add a replica. Can you try using the Collections API to
> create a collection?
>
> Starting Solr 5.0, the only supported way to create a new collection is
> via the Collections API. Creating a core would lead to a collection
> creation but that's not really supported. It was just something that was
> done when there were no Collections API.
>
>
> On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
>> I tried again with the following security.json, but the results were the
>> same:
>>
>> {
>> "authentication":{
>> "class":"solr.BasicAuthPlugin",
>> "credentials":{
>> "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
>> "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
>> 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
>> "":{"v":9}},
>> "authorization":{
>> "class":"solr.RuleBasedAuthorizationPlugin",
>> "user-role":{
>> "solr":[
>> "admin",
>> "read",
>> "xmpladmin",
>> "xmplgen",
>> "xmplsel"],
>> "solruser":[
>> "read",
>> "xmplgen",
>> "xmplsel"]},
>> "permissions":[
>> {
>> "name":"security-edit",
>> "role":"admin"},
>> {
>> "name":"xmpl_admin",
>> "collection":"xmpl",
>> "path":"/admin/*",
>> "role":"xmpladmin"},
>> {
>> "name":"xmpl_sel",
>> "collection":"xmpl",
>> "path":"/select/*",
>> "role":null},
>> {
>> "name":"all-admin",
>> "collection":null,
>> "path":"/*",
>> "role":"xmplgen"},
>> {
>> "name":"all-core-handlers",
>> "path":"/*",
>> "role":"xmplgen"}],
>> "":{"v":42}}}
>>
>> -----Original Message-----
>> From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> Sent: Thursday, November 19, 2015 1:46 PM
>> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>>
>> I note that the thread called "Security Problems" (most recent post by
>> Nobel Paul) seems like it may help with much of what I'm trying to do. I
>> will see to what extent that may help.
>>
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Anshum and Nobel,
I've downloaded 5.4, and this seems to be working so far
Thanks again
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Tuesday, December 01, 2015 12:52 AM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Hi Craig,
As part of my manual testing for the 5.3 RC, I tried out collection admin
request restriction and update restriction on a single node setup. I don't
have the manual test steps documented but it wasn't too intensive I'd
admit. I think the complications involved in stopping specific nodes and
bringing them back up stop us from testing the node restarts as part of the
automated tests but we should find a way and fix that.
I've just found another issue and opened SOLR-8355 for the same and it
involves the "update" permission.
As far as patching 5.3.1 go, it's involves more than just this one patch
and this patch alone wouldn't help you resolve this issue. You'd certainly
need the patch from SOLR-8167. Also, make sure you actually use the
'commit' and not the posted patch as the patch on SOLR-8167 is different
from the commit. I don't think you'd need anything other than those patches
and whatever comes from 8355 to have a patched 5.3.1.
Any help in testing this out would be awesome and thanks for reporting and
following up on the issues!
On Tue, Dec 1, 2015 at 6:09 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you, Anshum and Nobel, for your progress on SOLR-8326
>
> I have a couple questions to tide me over until 5.4 (hoping to test
> security.json a bit further while I wait).
>
> Given that the seven steps (tar xvzf solr-5.3.1.tgz; tar xvzf
> zookeeper-3.4.6.tar.gz; zkServer.sh start zoo_sample.cfg; zkcli.sh -zkhost
> localhost:2181 -cmd putfile /security.json ~/security.json; solr start -e
> cloud -z localhost:2181; solr stop -p 7574 & solr start -c -p 7574 -s
> "example/cloud/node2/solr" -z localhost:2181) demonstrate the problem, are
> there a similar set of steps by which one can load _some_ minimal
> security.json and still be able to stop & successfully restart one node of
> the cluster? (I am wondering what steps were used in the original testing
> of 5.3.1)
>
> Also, has it been verified that the SOLR-8326 patch resolves the
> ADDREPLICA bug in addition to the
> shutdown-&-restart-one-node-while-keeping-another-node-running bug?
>
> Also, would it make sense for me to download solr-5.3.1-src.tgz and (in a
> test environment) make the changes described in the latest attachment to
> SOLR-8326? Or would it be more advisable just to wait for 5.4? I don't know
> what may be involved in compiling a new solr.war from the source code.
>
> Thanks again
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, November 24, 2015 1:25 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> bq: I don't suppose there is an ETA for 5.4?
>
> Actually, 5.4 is probably in the works within the next month. I'm not
> the one cutting the
> release, but there's some rumors that a label will be cut this week,
> then the "usual"
> process is a week or two (sometimes more if bugs are flushed out) before
> the
> official release.
>
> Call it the first of the year for safety's sake, but that's a guess.
>
> Best,
> Erick
>
> On Tue, Nov 24, 2015 at 10:22 AM, Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov> wrote:
> > Thanks for the reply,
> >
> > I don't suppose there is an ETA for 5.4?
> >
> >
> > Thanks again
> >
> > -----Original Message-----
> ...
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
Hi Craig,
As part of my manual testing for the 5.3 RC, I tried out collection admin
request restriction and update restriction on a single node setup. I don't
have the manual test steps documented but it wasn't too intensive I'd
admit. I think the complications involved in stopping specific nodes and
bringing them back up stop us from testing the node restarts as part of the
automated tests but we should find a way and fix that.
I've just found another issue and opened SOLR-8355 for the same and it
involves the "update" permission.
As far as patching 5.3.1 go, it's involves more than just this one patch
and this patch alone wouldn't help you resolve this issue. You'd certainly
need the patch from SOLR-8167. Also, make sure you actually use the
'commit' and not the posted patch as the patch on SOLR-8167 is different
from the commit. I don't think you'd need anything other than those patches
and whatever comes from 8355 to have a patched 5.3.1.
Any help in testing this out would be awesome and thanks for reporting and
following up on the issues!
On Tue, Dec 1, 2015 at 6:09 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you, Anshum and Nobel, for your progress on SOLR-8326
>
> I have a couple questions to tide me over until 5.4 (hoping to test
> security.json a bit further while I wait).
>
> Given that the seven steps (tar xvzf solr-5.3.1.tgz; tar xvzf
> zookeeper-3.4.6.tar.gz; zkServer.sh start zoo_sample.cfg; zkcli.sh -zkhost
> localhost:2181 -cmd putfile /security.json ~/security.json; solr start -e
> cloud -z localhost:2181; solr stop -p 7574 & solr start -c -p 7574 -s
> "example/cloud/node2/solr" -z localhost:2181) demonstrate the problem, are
> there a similar set of steps by which one can load _some_ minimal
> security.json and still be able to stop & successfully restart one node of
> the cluster? (I am wondering what steps were used in the original testing
> of 5.3.1)
>
> Also, has it been verified that the SOLR-8326 patch resolves the
> ADDREPLICA bug in addition to the
> shutdown-&-restart-one-node-while-keeping-another-node-running bug?
>
> Also, would it make sense for me to download solr-5.3.1-src.tgz and (in a
> test environment) make the changes described in the latest attachment to
> SOLR-8326? Or would it be more advisable just to wait for 5.4? I don't know
> what may be involved in compiling a new solr.war from the source code.
>
> Thanks again
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, November 24, 2015 1:25 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> bq: I don't suppose there is an ETA for 5.4?
>
> Actually, 5.4 is probably in the works within the next month. I'm not
> the one cutting the
> release, but there's some rumors that a label will be cut this week,
> then the "usual"
> process is a week or two (sometimes more if bugs are flushed out) before
> the
> official release.
>
> Call it the first of the year for safety's sake, but that's a guess.
>
> Best,
> Erick
>
> On Tue, Nov 24, 2015 at 10:22 AM, Oakley, Craig (NIH/NLM/NCBI) [C]
> <cr...@nih.gov> wrote:
> > Thanks for the reply,
> >
> > I don't suppose there is an ETA for 5.4?
> >
> >
> > Thanks again
> >
> > -----Original Message-----
> ...
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thank you, Anshum and Nobel, for your progress on SOLR-8326
I have a couple questions to tide me over until 5.4 (hoping to test security.json a bit further while I wait).
Given that the seven steps (tar xvzf solr-5.3.1.tgz; tar xvzf zookeeper-3.4.6.tar.gz; zkServer.sh start zoo_sample.cfg; zkcli.sh -zkhost localhost:2181 -cmd putfile /security.json ~/security.json; solr start -e cloud -z localhost:2181; solr stop -p 7574 & solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181) demonstrate the problem, are there a similar set of steps by which one can load _some_ minimal security.json and still be able to stop & successfully restart one node of the cluster? (I am wondering what steps were used in the original testing of 5.3.1)
Also, has it been verified that the SOLR-8326 patch resolves the ADDREPLICA bug in addition to the shutdown-&-restart-one-node-while-keeping-another-node-running bug?
Also, would it make sense for me to download solr-5.3.1-src.tgz and (in a test environment) make the changes described in the latest attachment to SOLR-8326? Or would it be more advisable just to wait for 5.4? I don't know what may be involved in compiling a new solr.war from the source code.
Thanks again
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Tuesday, November 24, 2015 1:25 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
bq: I don't suppose there is an ETA for 5.4?
Actually, 5.4 is probably in the works within the next month. I'm not
the one cutting the
release, but there's some rumors that a label will be cut this week,
then the "usual"
process is a week or two (sometimes more if bugs are flushed out) before the
official release.
Call it the first of the year for safety's sake, but that's a guess.
Best,
Erick
On Tue, Nov 24, 2015 at 10:22 AM, Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov> wrote:
> Thanks for the reply,
>
> I don't suppose there is an ETA for 5.4?
>
>
> Thanks again
>
> -----Original Message-----
...
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Erick Erickson <er...@gmail.com>.
bq: I don't suppose there is an ETA for 5.4?
Actually, 5.4 is probably in the works within the next month. I'm not
the one cutting the
release, but there's some rumors that a label will be cut this week,
then the "usual"
process is a week or two (sometimes more if bugs are flushed out) before the
official release.
Call it the first of the year for safety's sake, but that's a guess.
Best,
Erick
On Tue, Nov 24, 2015 at 10:22 AM, Oakley, Craig (NIH/NLM/NCBI) [C]
<cr...@nih.gov> wrote:
> Thanks for the reply,
>
> I don't suppose there is an ETA for 5.4?
>
>
> Thanks again
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Tuesday, November 24, 2015 12:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Yes, it certainly is a PKI issue.
>
> On Tue, Nov 24, 2015 at 7:59 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
>> Thank you for the reply
>>
>> Trying those exact commands, I'm still getting the same issue
>> tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/solr-5.3.1.tgz
>> tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/zookeeper-3.4.6.tar.gz
>> cd zookeeper-3.4.6/
>> bin/zkServer.sh start zoo_sample.cfg
>> cd ..
>> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
>> -cmd putfile /security.json
>> PREVsolr-5.3.1/server/scripts/cloud-scripts/security.json
>> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
>> -cmd list
>> solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>> cd solr-5.3.1/
>> bin/solr stop -p 7574
>> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>> tail -f example/cloud/node2/logs/solr.log
>>
>> The -cmd list shows
>> / (2)
>> DATA:
>>
>> /zookeeper (1)
>> DATA:
>>
>> /security.json (0)
>> DATA:
>>
>> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":
>> {"solr":["admin"]}
>>
>> ,"permissions":[
>> {"name":"security-edit","role":"admin"}
>>
>> ]}}
>>
>>
>> While the output of tail contains
>> ERROR - 2015-11-24 10:45:54.796; [c:gettingstarted s:shard1 r:core_node4
>> x:gettingstarted_shard1_replica1] org.apache.solr.common.SolrException;
>> Error while trying to recover.:java.util.concurrent.ExecutionException:
>> org.apache.http.ParseException: Invalid content type:
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> Caused by: org.apache.http.ParseException: Invalid content type:
>> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> -----Original Message-----
>> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> Sent: Monday, November 23, 2015 7:24 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>>
>> Yes, I see the same issue. I'll update the JIRA and drill down. Thanks.
>>
>> On Mon, Nov 23, 2015 at 4:18 PM, Anshum Gupta <an...@anshumgupta.net>
>> wrote:
>>
>> > To restart solr, you should instead use something like:
>> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
>> > or
>> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>> >
>> > I've seen others report the same exception but never ran into this one
>> > myself. Let me try this out.
>> >
>> >
>> >
>> > On Mon, Nov 23, 2015 at 2:55 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> > craig.oakley@nih.gov> wrote:
>> >
>> >> FWIW
>> >>
>> >> I am getting fairly consistent results that if I follow the SOLR-8326
>> >> procedure just up through the step of "solr-5.3.1/bin/solr start -e
>> cloud
>> >> -z localhost:2181": if I then stop just one node (either "./solr stop -p
>> >> 7574" or "./solr stop -p 8983") and then restart that same node (using
>> the
>> >> command suggested by "solr-5.3.1/bin/solr start -e cloud -z
>> >> localhost:2181"), then the solr.log for the stopped-and-restarted node
>> gets
>> >> such stack traces as
>> >> ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3
>> >> x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException;
>> >> Error while trying to recover.:java.util.concurrent.ExecutionException:
>> >> org.apache.http.ParseException: Invalid content type:
>> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> >> at
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> >> at
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> >> at
>> >> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> >> Caused by: org.apache.http.ParseException: Invalid content type:
>> >> at
>> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> >> at
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> >> at
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> >> at
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >> at
>> >>
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> >> at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >>
>> >> While the node which stayed up the whole time starts getting such stack
>> >> traces as
>> >> ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3
>> >> x:gettingstarted_shard2_replica2]
>> >> org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
>> >> java.lang.NumberFormatException: For input string: "r?"
>> >> at
>> >>
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> >> at java.lang.Long.parseLong(Long.java:589)
>> >> at java.lang.Long.parseLong(Long.java:631)
>> >> at
>> >>
>> org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
>> >> at
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
>> >> at
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
>> >> at
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
>> >> at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> >> at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> >> at
>> >>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> >> at
>> >>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> >> at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> >> at
>> >>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> >> at
>> >>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> >> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> >> at
>> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> >> at
>> >>
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> >> at
>> >>
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> >> at
>> >>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> >> at
>> >>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> >> at java.lang.Thread.run(Thread.java:745)
>> >>
>> >> In this case the string is just "r?", but usually it is a longer string
>> >> of control characters.
>> >>
>> >> If I shutdown _both_ nodes and restart _one_, and then allow it to be
>> >> "Waiting until we see more replicas up" until it recognizes itself as
>> >> leader, and _then_ restart the other node -- in this case it
>> successfully
>> >> starts.
>> >>
>> >> Is there some necessary environment tweaking? The symptoms seem similar
>> >> whether I use the security.json from SOLR-8326 or the security.json from
>> >> the Wiki (with the comma repositioned).
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> >> Sent: Friday, November 20, 2015 6:59 PM
>> >> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> >> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>> >>
>> >> Thanks
>> >>
>> >> It seems to work when there is no security.json, so perhaps there's some
>> >> typo in the initial version.
>> >>
>> >> I notice that the version you sent is different from the documentation
>> at
>> >>
>> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
>> >> in that the Wiki version has "permissions" before "user-role": I also
>> >> notice that (at least as of right this moment) the Wiki version has a
>> comma
>> >> at the end of '"user-role":{"solr":"admin"},' even though it is at the
>> end:
>> >> and I notice that the Wiki version seems to lack a comma between the
>> >> "permissions" section and the "user-role" section. I just now also
>> noticed
>> >> that the version you sent has '"user-role":{"solr":["admin"]}' (with
>> square
>> >> brackets) whereas the Wiki does not have square brackets.
>> >>
>> >> The placement of the comma definitely looks wrong in the Wiki at the
>> >> moment (though perhaps someone might correct the Wiki before too long).
>> >> Other than that, I don’t know whether the order and/or the square
>> brackets
>> >> make a difference. I can try with different permutations.
>> >>
>> >> Thanks again
>> >>
>> >> P.S. for the record, the Wiki currently has
>> >> {
>> >> "authentication":{
>> >> "class":"solr.BasicAuthPlugin",
>> >> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> >> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>> >> },
>> >> "authorization":{
>> >> "class":"solr.RuleBasedAuthorizationPlugin",
>> >> "permissions":[{"name":"security-edit",
>> >> "role":"admin"}]
>> >> "user-role":{"solr":"admin"},
>> >> }}
>> >>
>> >> -----Original Message-----
>> >> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> >> Sent: Friday, November 20, 2015 6:18 PM
>> >> To: solr-user@lucene.apache.org
>> >> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>> >>
>> >> This seems unrelated and more like a user error somewhere. Can you just
>> >> follow the steps, without any security settings i.e. not even uploading
>> >> security.json and see if you still see this? Sorry, but I don't have
>> >> access
>> >> to the code right now, I try and look at this later tonight.
>> >>
>> >> On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> >> craig.oakley@nih.gov> wrote:
>> >>
>> >> > Thank you for opening SOLR-8326
>> >> >
>> >> > As a side note, in the procedure you listed, even before adding the
>> >> > collection-admin-edit authorization, I'm already hitting trouble:
>> >> stopping
>> >> > and restarting a node results in the following
>> >> >
>> >> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
>> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> >> > Publishing state of core solr8326_shard2_replica1 as recovering,
>> leader
>> >> is
>> >> > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and
>> I
>> >> am
>> >> > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
>> >> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
>> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController;
>> >> publishing
>> >> > state=recovering
>> >> > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
>> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> >> > Publishing state of core solr8326_shard1_replica1 as recovering,
>> leader
>> >> is
>> >> > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and
>> I
>> >> am
>> >> > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
>> >> > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
>> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController;
>> >> publishing
>> >> > state=recovering
>> >> > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
>> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> >> Sending
>> >> > prep recovery command to http://{IP-address-redacted}:8983/solr;
>> >> > WaitForState:
>> >> >
>> >>
>> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
>> >> > INFO - 2015-11-20 22:48:41.289; [ ]
>> >> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
>> >> > WatchedEvent state:SyncConnected type:NodeDataChanged
>> >> > path:/collections/solr8326/state.json for collection solr8326 has
>> >> occurred
>> >> > - updating... (live nodes size: 2)
>> >> > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
>> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> >> Sending
>> >> > prep recovery command to http://{IP-address-redacted}:8983/solr;
>> >> > WaitForState:
>> >> >
>> >>
>> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
>> >> > INFO - 2015-11-20 22:48:41.291; [ ]
>> >> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
>> >> to
>> >> > ver 25
>> >> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
>> >> > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException;
>> Error
>> >> > while trying to recover.:java.util.concurrent.ExecutionException:
>> >> > org.apache.http.ParseException: Invalid content type:
>> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> >> > at
>> >> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> >> > Caused by: org.apache.http.ParseException: Invalid content type:
>> >> > at
>> >> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> > at java.lang.Thread.run(Thread.java:745)
>> >> >
>> >> > INFO - 2015-11-20 22:48:41.298; [ ]
>> >> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
>> >> > WatchedEvent state:SyncConnected type:NodeDataChanged
>> >> > path:/collections/solr8326/state.json for collection solr8326 has
>> >> occurred
>> >> > - updating... (live nodes size: 2)
>> >> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
>> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> >> > Recovery failed - trying again... (4)
>> >> > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
>> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> Wait
>> >> > 32.0 seconds before trying to recover again (5)
>> >> > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
>> >> > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException;
>> Error
>> >> > while trying to recover.:java.util.concurrent.ExecutionException:
>> >> > org.apache.http.ParseException: Invalid content type:
>> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> >> > at
>> >> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> >> > Caused by: org.apache.http.ParseException: Invalid content type:
>> >> > at
>> >> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >> > at
>> >> >
>> >>
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> > at
>> >> >
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> > at java.lang.Thread.run(Thread.java:745)
>> >> >
>> >> > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
>> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> >> > Recovery failed - trying again... (4)
>> >> > INFO - 2015-11-20 22:48:41.318; [ ]
>> >> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
>> >> to
>> >> > ver 26
>> >> > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
>> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> Wait
>> >> > 32.0 seconds before trying to recover again (5)
>> >> >
>> >> >
>> >> > I would not be surprised if this were to be some unrelated issue (the
>> >> > symptoms are quite different)
>> >> >
>> >> >
>> >> >
>> >> > Thanks again
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> >> > Sent: Friday, November 20, 2015 1:31 PM
>> >> > To: solr-user@lucene.apache.org
>> >> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>> >> >
>> >> > Collections API were available before November of 2014, if that is
>> when
>> >> you
>> >> > took the class. However, it was only with Solr 5.0 (released in Feb
>> >> 2015)
>> >> > that the only supported mechanism to create a collection was
>> restricted
>> >> to
>> >> > Collections API.
>> >> >
>> >> > Here are the list of steps that you'd need to run to see that things
>> are
>> >> > fine for you without the read permission:
>> >> > * Untar and setup Solr, don't start it yet
>> >> > * Start clean zookeeper
>> >> > * Put the security.json in zk, without anything other than a
>> >> security-edit
>> >> > permission. Find the content of the file below. Upload it using your
>> >> own zk
>> >> > client or through the solr script:
>> >> > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost
>> >> localhost:2181
>> >> > -cmd putfile /security.json ~/security.json
>> >> >
>> >> > security.json:
>> >> >
>> >> >
>> >>
>> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> >> >
>> >> >
>> >>
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>> >> >
>> >> > * Start solr:
>> >> > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>> >> >
>> >> > You would need to key in a few things e.g. #nodes and ports, leave
>> them
>> >> at
>> >> > the default values of 2 nodes and 8983/7574, unless you want to run
>> >> Solr on
>> >> > a different port. Then let it create a default collection to just make
>> >> sure
>> >> > that everything works fine.
>> >> >
>> >> > * Add the collection-admin-edit command:
>> >> > > curl --user solr:SolrRocks
>> >> > http://localhost:8983/solr/admin/authorization
>> >> > -H 'Content-type:application/json' -d '{"set-permission" :
>> >> > {"name":"collection-admin-edit", "role":"admin"}}'
>> >> >
>> >> > At this point, everything should be working fine. Restarting the nodes
>> >> > should also work fine. You can try 2 things at this point:
>> >> > 1. Create a new collection with 1 shard and 1 replica and then try
>> >> adding a
>> >> > replica, here's how:
>> >> > > curl --user solr:SolrRocks
>> >> >
>> >> >
>> >>
>> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>> >> >
>> >> > > curl --user solr:SolrRocks
>> >> >
>> >> >
>> >>
>> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>> >> >
>> >> > This should work fine.
>> >> >
>> >> > 2. After this, try restarting the solr cluster. Here's how you can do
>> >> so,
>> >> > assuming you didn't change any of the defaults and you are running zk
>> on
>> >> > localhost:2181. If not, just change those values below:
>> >> > > bin/solr stop -all
>> >> >
>> >> > After this, check that Solr was actually stopped. I'd also suggest you
>> >> tail
>> >> > the logs on both nodes when they are coming up to see any errors, if
>> >> any.
>> >> > The logs would be here: example/cloud/node1/logs/solr.log
>> >> > and example/cloud/node2/logs/solr.log
>> >> >
>> >> > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z
>> >> localhost:2181
>> >> > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z
>> >> localhost:2181
>> >> >
>> >> > If you get to this checkpoint fine, try adding a read permission.
>> >> > Add a permission:
>> >> > > curl --user solr:SolrRocks
>> >> > http://localhost:8983/solr/admin/authorization
>> >> > -H 'Content-type:application/json' -d '{"set-permission" :
>> >> {"name":"read",
>> >> > "role":"read"}}'
>> >> >
>> >> > Add a user:
>> >> > > curl --user solr:SolrRocks
>> >> > http://localhost:8983/solr/admin/authentication
>> >> > -H 'Content-type:application/json' -d '{"set-user" :
>> >> > {"solrread":"solrRocks"}}'
>> >> >
>> >> > Assign a role to the user:
>> >> > >curl --user solr:SolrRocks
>> >> http://localhost:8983/solr/admin/authorization
>> >> > -H 'Content-type:application/json' -d '{"set-user-role" :
>> >> > {"solrread":["read"]}}'
>> >> >
>> >> > After this, you should start having issues with ADDREPLICA.
>> >> > Also, as you would at this point have a collection with a shard that
>> >> has a
>> >> > replication factor > 1 (remember the ADDREPLICA we did earlier), you
>> >> would
>> >> > have issues when you restart the cluster again using the steps I
>> >> mentioned
>> >> > above.
>> >> >
>> >> >
>> >> > Can you confirm this? I guess I'll just use this text to create a new
>> >> JIRA
>> >> > now.
>> >> >
>> >> >
>> >> > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> >> > craig.oakley@nih.gov> wrote:
>> >> >
>> >> > > Thank you again for the reply.
>> >> > >
>> >> > > Below is the Email I was about to send prior to your reply a moment
>> >> ago:
>> >> > > shall I try again without "read" in the security.json?
>> >> > >
>> >> > >
>> >> > >
>> >> > > The Collections API method was not discussed in the "Unleashed"
>> class
>> >> at
>> >> > > the conference in DC in 2014 (probably because it was not yet
>> >> available),
>> >> > > so I was using the method I knew.
>> >> > >
>> >> > > I have now tried again using admin/collections?action=CREATE (using
>> >> > > different port numbers to avoid confusion from the failed previous
>> >> > > attempts: the previously created nodes had been shutdown and their
>> >> > > core.properties files renamed so as not to be discovered), but the
>> >> > results
>> >> > > are the same:
>> >> > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> >> Starting
>> >> > > Replication Recovery.
>> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> Begin
>> >> > > buffering updates.
>> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting
>> to
>> >> > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
>> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> >> > Attempting
>> >> > > to replicate from http://
>> >> > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
>> >> > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
>> >> > while
>> >> > > trying to
>> >> > >
>> >> >
>> >>
>> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> >> > > Error from server at http://
>> >> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
>> >> > > Expected mime type application/octet-stream but got text/html.
>> <html>
>> >> > > <head>
>> >> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>> >> > > <title>Error 401 Unauthorized request, Response code: 401</title>
>> >> > > </head>
>> >> > > <body><h2>HTTP ERROR 401</h2>
>> >> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
>> >> > > <pre> Unauthorized request, Response code:
>> >> > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>> >> > >
>> >> > > </body>
>> >> > > </html>
>> >> > >
>> >> > > at
>> >> > >
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
>> >> > > at
>> >> > >
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
>> >> > > at
>> >> > >
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
>> >> > > at
>> >> > >
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
>> >> > > at
>> >> > >
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
>> >> > > at
>> >> > >
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
>> >> > > at
>> >> > >
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
>> >> > > at
>> >> > >
>> >> >
>> >>
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
>> >> > > at
>> >> > >
>> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> >> > >
>> >> > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
>> >> > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
>> >> > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> >> Recovery
>> >> > > failed - trying again... (2)
>> >> > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
>> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> Wait
>> >> 8.0
>> >> > > seconds before trying to recover again (3)
>> >> > >
>> >> > >
>> >> > > Below is a list of the steps I took.
>> >> > >
>> >> > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
>> >> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile
>> >> /security.json
>> >> > > ~/solr/security151119a.json
>> >> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
>> >> > > ../../solr/configsets/basic_configs/conf -confname xmpl3
>> >> > > cd ../../../bin/
>> >> > > ./solr -c -p 4695 -d
>> >> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
>> >> > -z
>> >> > > localhost:4545/solr/xmpl3 -s
>> >> > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
>> >> > > ./solr -c -p 4685 -d
>> >> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
>> >> > -z
>> >> > > localhost:4545/solr/xmpl3 -s
>> >> > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
>> >> > > curl -u solr:SolrRocks '
>> >> > >
>> >> >
>> >>
>> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
>> >> > > '
>> >> > > curl -u solr:SolrRocks '
>> >> > >
>> >> >
>> >>
>> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
>> >> > > '
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > Can you provide a list of steps to take in an out-of-the-box
>> directory
>> >> > > tree whereby ADDREPLICA _will_ work with security.json already in
>> >> place?
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > -----Original Message-----
>> >> > > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> >> > > Sent: Thursday, November 19, 2015 3:44 PM
>> >> > > To: solr-user@lucene.apache.org
>> >> > > Subject: Re: Re:Re: Implementing security.json is breaking
>> ADDREPLICA
>> >> > >
>> >> > > I'll try out what you did later in the day, as soon as I get time
>> but
>> >> why
>> >> > > exactly are you creating cores manually? Seems like you manually
>> >> create a
>> >> > > core and the try to add a replica. Can you try using the Collections
>> >> API
>> >> > to
>> >> > > create a collection?
>> >> > >
>> >> > > Starting Solr 5.0, the only supported way to create a new collection
>> >> is
>> >> > via
>> >> > > the Collections API. Creating a core would lead to a collection
>> >> creation
>> >> > > but that's not really supported. It was just something that was done
>> >> when
>> >> > > there were no Collections API.
>> >> > >
>> >> > >
>> >> > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> >> > > craig.oakley@nih.gov> wrote:
>> >> > >
>> >> > > > I tried again with the following security.json, but the results
>> were
>> >> > the
>> >> > > > same:
>> >> > > >
>> >> > > > {
>> >> > > > "authentication":{
>> >> > > > "class":"solr.BasicAuthPlugin",
>> >> > > > "credentials":{
>> >> > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> >> > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
>> >> > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
>> >> > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
>> >> > > > "":{"v":9}},
>> >> > > > "authorization":{
>> >> > > > "class":"solr.RuleBasedAuthorizationPlugin",
>> >> > > > "user-role":{
>> >> > > > "solr":[
>> >> > > > "admin",
>> >> > > > "read",
>> >> > > > "xmpladmin",
>> >> > > > "xmplgen",
>> >> > > > "xmplsel"],
>> >> > > > "solruser":[
>> >> > > > "read",
>> >> > > > "xmplgen",
>> >> > > > "xmplsel"]},
>> >> > > > "permissions":[
>> >> > > > {
>> >> > > > "name":"security-edit",
>> >> > > > "role":"admin"},
>> >> > > > {
>> >> > > > "name":"xmpl_admin",
>> >> > > > "collection":"xmpl",
>> >> > > > "path":"/admin/*",
>> >> > > > "role":"xmpladmin"},
>> >> > > > {
>> >> > > > "name":"xmpl_sel",
>> >> > > > "collection":"xmpl",
>> >> > > > "path":"/select/*",
>> >> > > > "role":null},
>> >> > > > {
>> >> > > > "name":"all-admin",
>> >> > > > "collection":null,
>> >> > > > "path":"/*",
>> >> > > > "role":"xmplgen"},
>> >> > > > {
>> >> > > > "name":"all-core-handlers",
>> >> > > > "path":"/*",
>> >> > > > "role":"xmplgen"}],
>> >> > > > "":{"v":42}}}
>> >> > > >
>> >> > > > -----Original Message-----
>> >> > > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> >> > > > Sent: Thursday, November 19, 2015 1:46 PM
>> >> > > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> >> > > > Subject: RE: Re:Re: Implementing security.json is breaking
>> >> ADDREPLICA
>> >> > > >
>> >> > > > I note that the thread called "Security Problems" (most recent
>> post
>> >> by
>> >> > > > Nobel Paul) seems like it may help with much of what I'm trying to
>> >> do.
>> >> > I
>> >> > > > will see to what extent that may help.
>> >> > > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Anshum Gupta
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Anshum Gupta
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Anshum Gupta
>> >>
>> >
>> >
>> >
>> > --
>> > Anshum Gupta
>> >
>>
>>
>>
>> --
>> Anshum Gupta
>>
>
>
>
> --
> Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thanks for the reply,
I don't suppose there is an ETA for 5.4?
Thanks again
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Tuesday, November 24, 2015 12:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Yes, it certainly is a PKI issue.
On Tue, Nov 24, 2015 at 7:59 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you for the reply
>
> Trying those exact commands, I'm still getting the same issue
> tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/solr-5.3.1.tgz
> tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/zookeeper-3.4.6.tar.gz
> cd zookeeper-3.4.6/
> bin/zkServer.sh start zoo_sample.cfg
> cd ..
> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd putfile /security.json
> PREVsolr-5.3.1/server/scripts/cloud-scripts/security.json
> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd list
> solr-5.3.1/bin/solr start -e cloud -z localhost:2181
> cd solr-5.3.1/
> bin/solr stop -p 7574
> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
> tail -f example/cloud/node2/logs/solr.log
>
> The -cmd list shows
> / (2)
> DATA:
>
> /zookeeper (1)
> DATA:
>
> /security.json (0)
> DATA:
>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":
> {"solr":["admin"]}
>
> ,"permissions":[
> {"name":"security-edit","role":"admin"}
>
> ]}}
>
>
> While the output of tail contains
> ERROR - 2015-11-24 10:45:54.796; [c:gettingstarted s:shard1 r:core_node4
> x:gettingstarted_shard1_replica1] org.apache.solr.common.SolrException;
> Error while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Monday, November 23, 2015 7:24 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Yes, I see the same issue. I'll update the JIRA and drill down. Thanks.
>
> On Mon, Nov 23, 2015 at 4:18 PM, Anshum Gupta <an...@anshumgupta.net>
> wrote:
>
> > To restart solr, you should instead use something like:
> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> > or
> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
> >
> > I've seen others report the same exception but never ran into this one
> > myself. Let me try this out.
> >
> >
> >
> > On Mon, Nov 23, 2015 at 2:55 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> >> FWIW
> >>
> >> I am getting fairly consistent results that if I follow the SOLR-8326
> >> procedure just up through the step of "solr-5.3.1/bin/solr start -e
> cloud
> >> -z localhost:2181": if I then stop just one node (either "./solr stop -p
> >> 7574" or "./solr stop -p 8983") and then restart that same node (using
> the
> >> command suggested by "solr-5.3.1/bin/solr start -e cloud -z
> >> localhost:2181"), then the solr.log for the stopped-and-restarted node
> gets
> >> such stack traces as
> >> ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3
> >> x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException;
> >> Error while trying to recover.:java.util.concurrent.ExecutionException:
> >> org.apache.http.ParseException: Invalid content type:
> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> at
> >>
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> >> at
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> >> at
> >> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> Caused by: org.apache.http.ParseException: Invalid content type:
> >> at
> org.apache.http.entity.ContentType.parse(ContentType.java:273)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> at
> >>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> While the node which stayed up the whole time starts getting such stack
> >> traces as
> >> ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3
> >> x:gettingstarted_shard2_replica2]
> >> org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
> >> java.lang.NumberFormatException: For input string: "r?"
> >> at
> >>
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >> at java.lang.Long.parseLong(Long.java:589)
> >> at java.lang.Long.parseLong(Long.java:631)
> >> at
> >>
> org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >> at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> >> at
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >> at
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >> at
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> In this case the string is just "r?", but usually it is a longer string
> >> of control characters.
> >>
> >> If I shutdown _both_ nodes and restart _one_, and then allow it to be
> >> "Waiting until we see more replicas up" until it recognizes itself as
> >> leader, and _then_ restart the other node -- in this case it
> successfully
> >> starts.
> >>
> >> Is there some necessary environment tweaking? The symptoms seem similar
> >> whether I use the security.json from SOLR-8326 or the security.json from
> >> the Wiki (with the comma repositioned).
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> >> Sent: Friday, November 20, 2015 6:59 PM
> >> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> >> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> >>
> >> Thanks
> >>
> >> It seems to work when there is no security.json, so perhaps there's some
> >> typo in the initial version.
> >>
> >> I notice that the version you sent is different from the documentation
> at
> >>
> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
> >> in that the Wiki version has "permissions" before "user-role": I also
> >> notice that (at least as of right this moment) the Wiki version has a
> comma
> >> at the end of '"user-role":{"solr":"admin"},' even though it is at the
> end:
> >> and I notice that the Wiki version seems to lack a comma between the
> >> "permissions" section and the "user-role" section. I just now also
> noticed
> >> that the version you sent has '"user-role":{"solr":["admin"]}' (with
> square
> >> brackets) whereas the Wiki does not have square brackets.
> >>
> >> The placement of the comma definitely looks wrong in the Wiki at the
> >> moment (though perhaps someone might correct the Wiki before too long).
> >> Other than that, I don’t know whether the order and/or the square
> brackets
> >> make a difference. I can try with different permutations.
> >>
> >> Thanks again
> >>
> >> P.S. for the record, the Wiki currently has
> >> {
> >> "authentication":{
> >> "class":"solr.BasicAuthPlugin",
> >> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
> >> },
> >> "authorization":{
> >> "class":"solr.RuleBasedAuthorizationPlugin",
> >> "permissions":[{"name":"security-edit",
> >> "role":"admin"}]
> >> "user-role":{"solr":"admin"},
> >> }}
> >>
> >> -----Original Message-----
> >> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> >> Sent: Friday, November 20, 2015 6:18 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >>
> >> This seems unrelated and more like a user error somewhere. Can you just
> >> follow the steps, without any security settings i.e. not even uploading
> >> security.json and see if you still see this? Sorry, but I don't have
> >> access
> >> to the code right now, I try and look at this later tonight.
> >>
> >> On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> >> craig.oakley@nih.gov> wrote:
> >>
> >> > Thank you for opening SOLR-8326
> >> >
> >> > As a side note, in the procedure you listed, even before adding the
> >> > collection-admin-edit authorization, I'm already hitting trouble:
> >> stopping
> >> > and restarting a node results in the following
> >> >
> >> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Publishing state of core solr8326_shard2_replica1 as recovering,
> leader
> >> is
> >> > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and
> I
> >> am
> >> > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> >> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController;
> >> publishing
> >> > state=recovering
> >> > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Publishing state of core solr8326_shard1_replica1 as recovering,
> leader
> >> is
> >> > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and
> I
> >> am
> >> > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> >> > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController;
> >> publishing
> >> > state=recovering
> >> > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> Sending
> >> > prep recovery command to http://{IP-address-redacted}:8983/solr;
> >> > WaitForState:
> >> >
> >>
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> >> > INFO - 2015-11-20 22:48:41.289; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> >> > WatchedEvent state:SyncConnected type:NodeDataChanged
> >> > path:/collections/solr8326/state.json for collection solr8326 has
> >> occurred
> >> > - updating... (live nodes size: 2)
> >> > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> Sending
> >> > prep recovery command to http://{IP-address-redacted}:8983/solr;
> >> > WaitForState:
> >> >
> >>
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> >> > INFO - 2015-11-20 22:48:41.291; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
> >> to
> >> > ver 25
> >> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException;
> Error
> >> > while trying to recover.:java.util.concurrent.ExecutionException:
> >> > org.apache.http.ParseException: Invalid content type:
> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> >> > at
> >> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> > Caused by: org.apache.http.ParseException: Invalid content type:
> >> > at
> >> org.apache.http.entity.ContentType.parse(ContentType.java:273)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > at
> >> >
> >>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >
> >> > INFO - 2015-11-20 22:48:41.298; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> >> > WatchedEvent state:SyncConnected type:NodeDataChanged
> >> > path:/collections/solr8326/state.json for collection solr8326 has
> >> occurred
> >> > - updating... (live nodes size: 2)
> >> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Recovery failed - trying again... (4)
> >> > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Wait
> >> > 32.0 seconds before trying to recover again (5)
> >> > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException;
> Error
> >> > while trying to recover.:java.util.concurrent.ExecutionException:
> >> > org.apache.http.ParseException: Invalid content type:
> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> >> > at
> >> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> > Caused by: org.apache.http.ParseException: Invalid content type:
> >> > at
> >> org.apache.http.entity.ContentType.parse(ContentType.java:273)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > at
> >> >
> >>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >
> >> > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Recovery failed - trying again... (4)
> >> > INFO - 2015-11-20 22:48:41.318; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
> >> to
> >> > ver 26
> >> > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Wait
> >> > 32.0 seconds before trying to recover again (5)
> >> >
> >> >
> >> > I would not be surprised if this were to be some unrelated issue (the
> >> > symptoms are quite different)
> >> >
> >> >
> >> >
> >> > Thanks again
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> >> > Sent: Friday, November 20, 2015 1:31 PM
> >> > To: solr-user@lucene.apache.org
> >> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >> >
> >> > Collections API were available before November of 2014, if that is
> when
> >> you
> >> > took the class. However, it was only with Solr 5.0 (released in Feb
> >> 2015)
> >> > that the only supported mechanism to create a collection was
> restricted
> >> to
> >> > Collections API.
> >> >
> >> > Here are the list of steps that you'd need to run to see that things
> are
> >> > fine for you without the read permission:
> >> > * Untar and setup Solr, don't start it yet
> >> > * Start clean zookeeper
> >> > * Put the security.json in zk, without anything other than a
> >> security-edit
> >> > permission. Find the content of the file below. Upload it using your
> >> own zk
> >> > client or through the solr script:
> >> > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost
> >> localhost:2181
> >> > -cmd putfile /security.json ~/security.json
> >> >
> >> > security.json:
> >> >
> >> >
> >>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >> >
> >> >
> >>
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
> >> >
> >> > * Start solr:
> >> > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
> >> >
> >> > You would need to key in a few things e.g. #nodes and ports, leave
> them
> >> at
> >> > the default values of 2 nodes and 8983/7574, unless you want to run
> >> Solr on
> >> > a different port. Then let it create a default collection to just make
> >> sure
> >> > that everything works fine.
> >> >
> >> > * Add the collection-admin-edit command:
> >> > > curl --user solr:SolrRocks
> >> > http://localhost:8983/solr/admin/authorization
> >> > -H 'Content-type:application/json' -d '{"set-permission" :
> >> > {"name":"collection-admin-edit", "role":"admin"}}'
> >> >
> >> > At this point, everything should be working fine. Restarting the nodes
> >> > should also work fine. You can try 2 things at this point:
> >> > 1. Create a new collection with 1 shard and 1 replica and then try
> >> adding a
> >> > replica, here's how:
> >> > > curl --user solr:SolrRocks
> >> >
> >> >
> >>
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
> >> >
> >> > > curl --user solr:SolrRocks
> >> >
> >> >
> >>
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
> >> >
> >> > This should work fine.
> >> >
> >> > 2. After this, try restarting the solr cluster. Here's how you can do
> >> so,
> >> > assuming you didn't change any of the defaults and you are running zk
> on
> >> > localhost:2181. If not, just change those values below:
> >> > > bin/solr stop -all
> >> >
> >> > After this, check that Solr was actually stopped. I'd also suggest you
> >> tail
> >> > the logs on both nodes when they are coming up to see any errors, if
> >> any.
> >> > The logs would be here: example/cloud/node1/logs/solr.log
> >> > and example/cloud/node2/logs/solr.log
> >> >
> >> > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z
> >> localhost:2181
> >> > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z
> >> localhost:2181
> >> >
> >> > If you get to this checkpoint fine, try adding a read permission.
> >> > Add a permission:
> >> > > curl --user solr:SolrRocks
> >> > http://localhost:8983/solr/admin/authorization
> >> > -H 'Content-type:application/json' -d '{"set-permission" :
> >> {"name":"read",
> >> > "role":"read"}}'
> >> >
> >> > Add a user:
> >> > > curl --user solr:SolrRocks
> >> > http://localhost:8983/solr/admin/authentication
> >> > -H 'Content-type:application/json' -d '{"set-user" :
> >> > {"solrread":"solrRocks"}}'
> >> >
> >> > Assign a role to the user:
> >> > >curl --user solr:SolrRocks
> >> http://localhost:8983/solr/admin/authorization
> >> > -H 'Content-type:application/json' -d '{"set-user-role" :
> >> > {"solrread":["read"]}}'
> >> >
> >> > After this, you should start having issues with ADDREPLICA.
> >> > Also, as you would at this point have a collection with a shard that
> >> has a
> >> > replication factor > 1 (remember the ADDREPLICA we did earlier), you
> >> would
> >> > have issues when you restart the cluster again using the steps I
> >> mentioned
> >> > above.
> >> >
> >> >
> >> > Can you confirm this? I guess I'll just use this text to create a new
> >> JIRA
> >> > now.
> >> >
> >> >
> >> > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> >> > craig.oakley@nih.gov> wrote:
> >> >
> >> > > Thank you again for the reply.
> >> > >
> >> > > Below is the Email I was about to send prior to your reply a moment
> >> ago:
> >> > > shall I try again without "read" in the security.json?
> >> > >
> >> > >
> >> > >
> >> > > The Collections API method was not discussed in the "Unleashed"
> class
> >> at
> >> > > the conference in DC in 2014 (probably because it was not yet
> >> available),
> >> > > so I was using the method I knew.
> >> > >
> >> > > I have now tried again using admin/collections?action=CREATE (using
> >> > > different port numbers to avoid confusion from the failed previous
> >> > > attempts: the previously created nodes had been shutdown and their
> >> > > core.properties files renamed so as not to be discovered), but the
> >> > results
> >> > > are the same:
> >> > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> >> Starting
> >> > > Replication Recovery.
> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Begin
> >> > > buffering updates.
> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting
> to
> >> > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> >> > Attempting
> >> > > to replicate from http://
> >> > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> >> > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> >> > while
> >> > > trying to
> >> > >
> >> >
> >>
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> >> > > Error from server at http://
> >> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> >> > > Expected mime type application/octet-stream but got text/html.
> <html>
> >> > > <head>
> >> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> >> > > <title>Error 401 Unauthorized request, Response code: 401</title>
> >> > > </head>
> >> > > <body><h2>HTTP ERROR 401</h2>
> >> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> >> > > <pre> Unauthorized request, Response code:
> >> > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> >> > >
> >> > > </body>
> >> > > </html>
> >> > >
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> >> > > at
> >> > >
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> >> > > at
> >> > >
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> >> > > at
> >> > >
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> > >
> >> > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> >> > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> >> > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> >> Recovery
> >> > > failed - trying again... (2)
> >> > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Wait
> >> 8.0
> >> > > seconds before trying to recover again (3)
> >> > >
> >> > >
> >> > > Below is a list of the steps I took.
> >> > >
> >> > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> >> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile
> >> /security.json
> >> > > ~/solr/security151119a.json
> >> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> >> > > ../../solr/configsets/basic_configs/conf -confname xmpl3
> >> > > cd ../../../bin/
> >> > > ./solr -c -p 4695 -d
> >> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> >> > -z
> >> > > localhost:4545/solr/xmpl3 -s
> >> > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> >> > > ./solr -c -p 4685 -d
> >> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> >> > -z
> >> > > localhost:4545/solr/xmpl3 -s
> >> > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> >> > > curl -u solr:SolrRocks '
> >> > >
> >> >
> >>
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> >> > > '
> >> > > curl -u solr:SolrRocks '
> >> > >
> >> >
> >>
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> >> > > '
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Can you provide a list of steps to take in an out-of-the-box
> directory
> >> > > tree whereby ADDREPLICA _will_ work with security.json already in
> >> place?
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > -----Original Message-----
> >> > > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> >> > > Sent: Thursday, November 19, 2015 3:44 PM
> >> > > To: solr-user@lucene.apache.org
> >> > > Subject: Re: Re:Re: Implementing security.json is breaking
> ADDREPLICA
> >> > >
> >> > > I'll try out what you did later in the day, as soon as I get time
> but
> >> why
> >> > > exactly are you creating cores manually? Seems like you manually
> >> create a
> >> > > core and the try to add a replica. Can you try using the Collections
> >> API
> >> > to
> >> > > create a collection?
> >> > >
> >> > > Starting Solr 5.0, the only supported way to create a new collection
> >> is
> >> > via
> >> > > the Collections API. Creating a core would lead to a collection
> >> creation
> >> > > but that's not really supported. It was just something that was done
> >> when
> >> > > there were no Collections API.
> >> > >
> >> > >
> >> > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> >> > > craig.oakley@nih.gov> wrote:
> >> > >
> >> > > > I tried again with the following security.json, but the results
> were
> >> > the
> >> > > > same:
> >> > > >
> >> > > > {
> >> > > > "authentication":{
> >> > > > "class":"solr.BasicAuthPlugin",
> >> > > > "credentials":{
> >> > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >> > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> >> > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> >> > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> >> > > > "":{"v":9}},
> >> > > > "authorization":{
> >> > > > "class":"solr.RuleBasedAuthorizationPlugin",
> >> > > > "user-role":{
> >> > > > "solr":[
> >> > > > "admin",
> >> > > > "read",
> >> > > > "xmpladmin",
> >> > > > "xmplgen",
> >> > > > "xmplsel"],
> >> > > > "solruser":[
> >> > > > "read",
> >> > > > "xmplgen",
> >> > > > "xmplsel"]},
> >> > > > "permissions":[
> >> > > > {
> >> > > > "name":"security-edit",
> >> > > > "role":"admin"},
> >> > > > {
> >> > > > "name":"xmpl_admin",
> >> > > > "collection":"xmpl",
> >> > > > "path":"/admin/*",
> >> > > > "role":"xmpladmin"},
> >> > > > {
> >> > > > "name":"xmpl_sel",
> >> > > > "collection":"xmpl",
> >> > > > "path":"/select/*",
> >> > > > "role":null},
> >> > > > {
> >> > > > "name":"all-admin",
> >> > > > "collection":null,
> >> > > > "path":"/*",
> >> > > > "role":"xmplgen"},
> >> > > > {
> >> > > > "name":"all-core-handlers",
> >> > > > "path":"/*",
> >> > > > "role":"xmplgen"}],
> >> > > > "":{"v":42}}}
> >> > > >
> >> > > > -----Original Message-----
> >> > > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> >> > > > Sent: Thursday, November 19, 2015 1:46 PM
> >> > > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> >> > > > Subject: RE: Re:Re: Implementing security.json is breaking
> >> ADDREPLICA
> >> > > >
> >> > > > I note that the thread called "Security Problems" (most recent
> post
> >> by
> >> > > > Nobel Paul) seems like it may help with much of what I'm trying to
> >> do.
> >> > I
> >> > > > will see to what extent that may help.
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Anshum Gupta
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Anshum Gupta
> >> >
> >>
> >>
> >>
> >> --
> >> Anshum Gupta
> >>
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
Yes, it certainly is a PKI issue.
On Tue, Nov 24, 2015 at 7:59 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you for the reply
>
> Trying those exact commands, I'm still getting the same issue
> tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/solr-5.3.1.tgz
> tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/zookeeper-3.4.6.tar.gz
> cd zookeeper-3.4.6/
> bin/zkServer.sh start zoo_sample.cfg
> cd ..
> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd putfile /security.json
> PREVsolr-5.3.1/server/scripts/cloud-scripts/security.json
> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd list
> solr-5.3.1/bin/solr start -e cloud -z localhost:2181
> cd solr-5.3.1/
> bin/solr stop -p 7574
> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
> tail -f example/cloud/node2/logs/solr.log
>
> The -cmd list shows
> / (2)
> DATA:
>
> /zookeeper (1)
> DATA:
>
> /security.json (0)
> DATA:
>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":
> {"solr":["admin"]}
>
> ,"permissions":[
> {"name":"security-edit","role":"admin"}
>
> ]}}
>
>
> While the output of tail contains
> ERROR - 2015-11-24 10:45:54.796; [c:gettingstarted s:shard1 r:core_node4
> x:gettingstarted_shard1_replica1] org.apache.solr.common.SolrException;
> Error while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Monday, November 23, 2015 7:24 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Yes, I see the same issue. I'll update the JIRA and drill down. Thanks.
>
> On Mon, Nov 23, 2015 at 4:18 PM, Anshum Gupta <an...@anshumgupta.net>
> wrote:
>
> > To restart solr, you should instead use something like:
> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> > or
> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
> >
> > I've seen others report the same exception but never ran into this one
> > myself. Let me try this out.
> >
> >
> >
> > On Mon, Nov 23, 2015 at 2:55 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> >> FWIW
> >>
> >> I am getting fairly consistent results that if I follow the SOLR-8326
> >> procedure just up through the step of "solr-5.3.1/bin/solr start -e
> cloud
> >> -z localhost:2181": if I then stop just one node (either "./solr stop -p
> >> 7574" or "./solr stop -p 8983") and then restart that same node (using
> the
> >> command suggested by "solr-5.3.1/bin/solr start -e cloud -z
> >> localhost:2181"), then the solr.log for the stopped-and-restarted node
> gets
> >> such stack traces as
> >> ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3
> >> x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException;
> >> Error while trying to recover.:java.util.concurrent.ExecutionException:
> >> org.apache.http.ParseException: Invalid content type:
> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> at
> >>
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> >> at
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> >> at
> >> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> Caused by: org.apache.http.ParseException: Invalid content type:
> >> at
> org.apache.http.entity.ContentType.parse(ContentType.java:273)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> at
> >>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> While the node which stayed up the whole time starts getting such stack
> >> traces as
> >> ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3
> >> x:gettingstarted_shard2_replica2]
> >> org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
> >> java.lang.NumberFormatException: For input string: "r?"
> >> at
> >>
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >> at java.lang.Long.parseLong(Long.java:589)
> >> at java.lang.Long.parseLong(Long.java:631)
> >> at
> >>
> org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
> >> at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> >> at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> >> at
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >> at
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >> at
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> In this case the string is just "r?", but usually it is a longer string
> >> of control characters.
> >>
> >> If I shutdown _both_ nodes and restart _one_, and then allow it to be
> >> "Waiting until we see more replicas up" until it recognizes itself as
> >> leader, and _then_ restart the other node -- in this case it
> successfully
> >> starts.
> >>
> >> Is there some necessary environment tweaking? The symptoms seem similar
> >> whether I use the security.json from SOLR-8326 or the security.json from
> >> the Wiki (with the comma repositioned).
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> >> Sent: Friday, November 20, 2015 6:59 PM
> >> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> >> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> >>
> >> Thanks
> >>
> >> It seems to work when there is no security.json, so perhaps there's some
> >> typo in the initial version.
> >>
> >> I notice that the version you sent is different from the documentation
> at
> >>
> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
> >> in that the Wiki version has "permissions" before "user-role": I also
> >> notice that (at least as of right this moment) the Wiki version has a
> comma
> >> at the end of '"user-role":{"solr":"admin"},' even though it is at the
> end:
> >> and I notice that the Wiki version seems to lack a comma between the
> >> "permissions" section and the "user-role" section. I just now also
> noticed
> >> that the version you sent has '"user-role":{"solr":["admin"]}' (with
> square
> >> brackets) whereas the Wiki does not have square brackets.
> >>
> >> The placement of the comma definitely looks wrong in the Wiki at the
> >> moment (though perhaps someone might correct the Wiki before too long).
> >> Other than that, I don’t know whether the order and/or the square
> brackets
> >> make a difference. I can try with different permutations.
> >>
> >> Thanks again
> >>
> >> P.S. for the record, the Wiki currently has
> >> {
> >> "authentication":{
> >> "class":"solr.BasicAuthPlugin",
> >> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
> >> },
> >> "authorization":{
> >> "class":"solr.RuleBasedAuthorizationPlugin",
> >> "permissions":[{"name":"security-edit",
> >> "role":"admin"}]
> >> "user-role":{"solr":"admin"},
> >> }}
> >>
> >> -----Original Message-----
> >> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> >> Sent: Friday, November 20, 2015 6:18 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >>
> >> This seems unrelated and more like a user error somewhere. Can you just
> >> follow the steps, without any security settings i.e. not even uploading
> >> security.json and see if you still see this? Sorry, but I don't have
> >> access
> >> to the code right now, I try and look at this later tonight.
> >>
> >> On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> >> craig.oakley@nih.gov> wrote:
> >>
> >> > Thank you for opening SOLR-8326
> >> >
> >> > As a side note, in the procedure you listed, even before adding the
> >> > collection-admin-edit authorization, I'm already hitting trouble:
> >> stopping
> >> > and restarting a node results in the following
> >> >
> >> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Publishing state of core solr8326_shard2_replica1 as recovering,
> leader
> >> is
> >> > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and
> I
> >> am
> >> > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> >> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController;
> >> publishing
> >> > state=recovering
> >> > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Publishing state of core solr8326_shard1_replica1 as recovering,
> leader
> >> is
> >> > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and
> I
> >> am
> >> > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> >> > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController;
> >> publishing
> >> > state=recovering
> >> > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> Sending
> >> > prep recovery command to http://{IP-address-redacted}:8983/solr;
> >> > WaitForState:
> >> >
> >>
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> >> > INFO - 2015-11-20 22:48:41.289; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> >> > WatchedEvent state:SyncConnected type:NodeDataChanged
> >> > path:/collections/solr8326/state.json for collection solr8326 has
> >> occurred
> >> > - updating... (live nodes size: 2)
> >> > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> Sending
> >> > prep recovery command to http://{IP-address-redacted}:8983/solr;
> >> > WaitForState:
> >> >
> >>
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> >> > INFO - 2015-11-20 22:48:41.291; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
> >> to
> >> > ver 25
> >> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException;
> Error
> >> > while trying to recover.:java.util.concurrent.ExecutionException:
> >> > org.apache.http.ParseException: Invalid content type:
> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> >> > at
> >> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> > Caused by: org.apache.http.ParseException: Invalid content type:
> >> > at
> >> org.apache.http.entity.ContentType.parse(ContentType.java:273)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > at
> >> >
> >>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >
> >> > INFO - 2015-11-20 22:48:41.298; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> >> > WatchedEvent state:SyncConnected type:NodeDataChanged
> >> > path:/collections/solr8326/state.json for collection solr8326 has
> >> occurred
> >> > - updating... (live nodes size: 2)
> >> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Recovery failed - trying again... (4)
> >> > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> >> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Wait
> >> > 32.0 seconds before trying to recover again (5)
> >> > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException;
> Error
> >> > while trying to recover.:java.util.concurrent.ExecutionException:
> >> > org.apache.http.ParseException: Invalid content type:
> >> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> >> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> >> > at
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> >> > at
> >> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> > Caused by: org.apache.http.ParseException: Invalid content type:
> >> > at
> >> org.apache.http.entity.ContentType.parse(ContentType.java:273)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> >> > at
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > at
> >> >
> >>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >
> >> > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> >> > Recovery failed - trying again... (4)
> >> > INFO - 2015-11-20 22:48:41.318; [ ]
> >> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
> >> to
> >> > ver 26
> >> > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> >> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Wait
> >> > 32.0 seconds before trying to recover again (5)
> >> >
> >> >
> >> > I would not be surprised if this were to be some unrelated issue (the
> >> > symptoms are quite different)
> >> >
> >> >
> >> >
> >> > Thanks again
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> >> > Sent: Friday, November 20, 2015 1:31 PM
> >> > To: solr-user@lucene.apache.org
> >> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >> >
> >> > Collections API were available before November of 2014, if that is
> when
> >> you
> >> > took the class. However, it was only with Solr 5.0 (released in Feb
> >> 2015)
> >> > that the only supported mechanism to create a collection was
> restricted
> >> to
> >> > Collections API.
> >> >
> >> > Here are the list of steps that you'd need to run to see that things
> are
> >> > fine for you without the read permission:
> >> > * Untar and setup Solr, don't start it yet
> >> > * Start clean zookeeper
> >> > * Put the security.json in zk, without anything other than a
> >> security-edit
> >> > permission. Find the content of the file below. Upload it using your
> >> own zk
> >> > client or through the solr script:
> >> > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost
> >> localhost:2181
> >> > -cmd putfile /security.json ~/security.json
> >> >
> >> > security.json:
> >> >
> >> >
> >>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >> >
> >> >
> >>
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
> >> >
> >> > * Start solr:
> >> > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
> >> >
> >> > You would need to key in a few things e.g. #nodes and ports, leave
> them
> >> at
> >> > the default values of 2 nodes and 8983/7574, unless you want to run
> >> Solr on
> >> > a different port. Then let it create a default collection to just make
> >> sure
> >> > that everything works fine.
> >> >
> >> > * Add the collection-admin-edit command:
> >> > > curl --user solr:SolrRocks
> >> > http://localhost:8983/solr/admin/authorization
> >> > -H 'Content-type:application/json' -d '{"set-permission" :
> >> > {"name":"collection-admin-edit", "role":"admin"}}'
> >> >
> >> > At this point, everything should be working fine. Restarting the nodes
> >> > should also work fine. You can try 2 things at this point:
> >> > 1. Create a new collection with 1 shard and 1 replica and then try
> >> adding a
> >> > replica, here's how:
> >> > > curl --user solr:SolrRocks
> >> >
> >> >
> >>
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
> >> >
> >> > > curl --user solr:SolrRocks
> >> >
> >> >
> >>
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
> >> >
> >> > This should work fine.
> >> >
> >> > 2. After this, try restarting the solr cluster. Here's how you can do
> >> so,
> >> > assuming you didn't change any of the defaults and you are running zk
> on
> >> > localhost:2181. If not, just change those values below:
> >> > > bin/solr stop -all
> >> >
> >> > After this, check that Solr was actually stopped. I'd also suggest you
> >> tail
> >> > the logs on both nodes when they are coming up to see any errors, if
> >> any.
> >> > The logs would be here: example/cloud/node1/logs/solr.log
> >> > and example/cloud/node2/logs/solr.log
> >> >
> >> > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z
> >> localhost:2181
> >> > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z
> >> localhost:2181
> >> >
> >> > If you get to this checkpoint fine, try adding a read permission.
> >> > Add a permission:
> >> > > curl --user solr:SolrRocks
> >> > http://localhost:8983/solr/admin/authorization
> >> > -H 'Content-type:application/json' -d '{"set-permission" :
> >> {"name":"read",
> >> > "role":"read"}}'
> >> >
> >> > Add a user:
> >> > > curl --user solr:SolrRocks
> >> > http://localhost:8983/solr/admin/authentication
> >> > -H 'Content-type:application/json' -d '{"set-user" :
> >> > {"solrread":"solrRocks"}}'
> >> >
> >> > Assign a role to the user:
> >> > >curl --user solr:SolrRocks
> >> http://localhost:8983/solr/admin/authorization
> >> > -H 'Content-type:application/json' -d '{"set-user-role" :
> >> > {"solrread":["read"]}}'
> >> >
> >> > After this, you should start having issues with ADDREPLICA.
> >> > Also, as you would at this point have a collection with a shard that
> >> has a
> >> > replication factor > 1 (remember the ADDREPLICA we did earlier), you
> >> would
> >> > have issues when you restart the cluster again using the steps I
> >> mentioned
> >> > above.
> >> >
> >> >
> >> > Can you confirm this? I guess I'll just use this text to create a new
> >> JIRA
> >> > now.
> >> >
> >> >
> >> > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> >> > craig.oakley@nih.gov> wrote:
> >> >
> >> > > Thank you again for the reply.
> >> > >
> >> > > Below is the Email I was about to send prior to your reply a moment
> >> ago:
> >> > > shall I try again without "read" in the security.json?
> >> > >
> >> > >
> >> > >
> >> > > The Collections API method was not discussed in the "Unleashed"
> class
> >> at
> >> > > the conference in DC in 2014 (probably because it was not yet
> >> available),
> >> > > so I was using the method I knew.
> >> > >
> >> > > I have now tried again using admin/collections?action=CREATE (using
> >> > > different port numbers to avoid confusion from the failed previous
> >> > > attempts: the previously created nodes had been shutdown and their
> >> > > core.properties files renamed so as not to be discovered), but the
> >> > results
> >> > > are the same:
> >> > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> >> Starting
> >> > > Replication Recovery.
> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Begin
> >> > > buffering updates.
> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting
> to
> >> > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> >> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> >> > Attempting
> >> > > to replicate from http://
> >> > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> >> > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> >> > while
> >> > > trying to
> >> > >
> >> >
> >>
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> >> > > Error from server at http://
> >> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> >> > > Expected mime type application/octet-stream but got text/html.
> <html>
> >> > > <head>
> >> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> >> > > <title>Error 401 Unauthorized request, Response code: 401</title>
> >> > > </head>
> >> > > <body><h2>HTTP ERROR 401</h2>
> >> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> >> > > <pre> Unauthorized request, Response code:
> >> > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> >> > >
> >> > > </body>
> >> > > </html>
> >> > >
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> >> > > at
> >> > >
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> >> > > at
> >> > >
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> >> > > at
> >> > >
> >> >
> >>
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> >> > > at
> >> > >
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >> > >
> >> > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> >> > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> >> > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> >> Recovery
> >> > > failed - trying again... (2)
> >> > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> >> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Wait
> >> 8.0
> >> > > seconds before trying to recover again (3)
> >> > >
> >> > >
> >> > > Below is a list of the steps I took.
> >> > >
> >> > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> >> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile
> >> /security.json
> >> > > ~/solr/security151119a.json
> >> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> >> > > ../../solr/configsets/basic_configs/conf -confname xmpl3
> >> > > cd ../../../bin/
> >> > > ./solr -c -p 4695 -d
> >> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> >> > -z
> >> > > localhost:4545/solr/xmpl3 -s
> >> > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> >> > > ./solr -c -p 4685 -d
> >> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> >> > -z
> >> > > localhost:4545/solr/xmpl3 -s
> >> > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> >> > > curl -u solr:SolrRocks '
> >> > >
> >> >
> >>
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> >> > > '
> >> > > curl -u solr:SolrRocks '
> >> > >
> >> >
> >>
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> >> > > '
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Can you provide a list of steps to take in an out-of-the-box
> directory
> >> > > tree whereby ADDREPLICA _will_ work with security.json already in
> >> place?
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > -----Original Message-----
> >> > > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> >> > > Sent: Thursday, November 19, 2015 3:44 PM
> >> > > To: solr-user@lucene.apache.org
> >> > > Subject: Re: Re:Re: Implementing security.json is breaking
> ADDREPLICA
> >> > >
> >> > > I'll try out what you did later in the day, as soon as I get time
> but
> >> why
> >> > > exactly are you creating cores manually? Seems like you manually
> >> create a
> >> > > core and the try to add a replica. Can you try using the Collections
> >> API
> >> > to
> >> > > create a collection?
> >> > >
> >> > > Starting Solr 5.0, the only supported way to create a new collection
> >> is
> >> > via
> >> > > the Collections API. Creating a core would lead to a collection
> >> creation
> >> > > but that's not really supported. It was just something that was done
> >> when
> >> > > there were no Collections API.
> >> > >
> >> > >
> >> > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> >> > > craig.oakley@nih.gov> wrote:
> >> > >
> >> > > > I tried again with the following security.json, but the results
> were
> >> > the
> >> > > > same:
> >> > > >
> >> > > > {
> >> > > > "authentication":{
> >> > > > "class":"solr.BasicAuthPlugin",
> >> > > > "credentials":{
> >> > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >> > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> >> > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> >> > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> >> > > > "":{"v":9}},
> >> > > > "authorization":{
> >> > > > "class":"solr.RuleBasedAuthorizationPlugin",
> >> > > > "user-role":{
> >> > > > "solr":[
> >> > > > "admin",
> >> > > > "read",
> >> > > > "xmpladmin",
> >> > > > "xmplgen",
> >> > > > "xmplsel"],
> >> > > > "solruser":[
> >> > > > "read",
> >> > > > "xmplgen",
> >> > > > "xmplsel"]},
> >> > > > "permissions":[
> >> > > > {
> >> > > > "name":"security-edit",
> >> > > > "role":"admin"},
> >> > > > {
> >> > > > "name":"xmpl_admin",
> >> > > > "collection":"xmpl",
> >> > > > "path":"/admin/*",
> >> > > > "role":"xmpladmin"},
> >> > > > {
> >> > > > "name":"xmpl_sel",
> >> > > > "collection":"xmpl",
> >> > > > "path":"/select/*",
> >> > > > "role":null},
> >> > > > {
> >> > > > "name":"all-admin",
> >> > > > "collection":null,
> >> > > > "path":"/*",
> >> > > > "role":"xmplgen"},
> >> > > > {
> >> > > > "name":"all-core-handlers",
> >> > > > "path":"/*",
> >> > > > "role":"xmplgen"}],
> >> > > > "":{"v":42}}}
> >> > > >
> >> > > > -----Original Message-----
> >> > > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> >> > > > Sent: Thursday, November 19, 2015 1:46 PM
> >> > > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> >> > > > Subject: RE: Re:Re: Implementing security.json is breaking
> >> ADDREPLICA
> >> > > >
> >> > > > I note that the thread called "Security Problems" (most recent
> post
> >> by
> >> > > > Nobel Paul) seems like it may help with much of what I'm trying to
> >> do.
> >> > I
> >> > > > will see to what extent that may help.
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Anshum Gupta
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Anshum Gupta
> >> >
> >>
> >>
> >>
> >> --
> >> Anshum Gupta
> >>
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thank you for the reply
Trying those exact commands, I'm still getting the same issue
tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/solr-5.3.1.tgz
tar xvzf /net/sybdev11/export/home/sybase/Distr/Solr/zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6/
bin/zkServer.sh start zoo_sample.cfg
cd ..
solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd putfile /security.json PREVsolr-5.3.1/server/scripts/cloud-scripts/security.json
solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd list
solr-5.3.1/bin/solr start -e cloud -z localhost:2181
cd solr-5.3.1/
bin/solr stop -p 7574
bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
tail -f example/cloud/node2/logs/solr.log
The -cmd list shows
/ (2)
DATA:
/zookeeper (1)
DATA:
/security.json (0)
DATA:
{"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":
{"solr":["admin"]}
,"permissions":[
{"name":"security-edit","role":"admin"}
]}}
While the output of tail contains
ERROR - 2015-11-24 10:45:54.796; [c:gettingstarted s:shard1 r:core_node4 x:gettingstarted_shard1_replica1] org.apache.solr.common.SolrException; Error while trying to recover.:java.util.concurrent.ExecutionException: org.apache.http.ParseException: Invalid content type:
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Caused by: org.apache.http.ParseException: Invalid content type:
at org.apache.http.entity.ContentType.parse(ContentType.java:273)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Monday, November 23, 2015 7:24 PM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Yes, I see the same issue. I'll update the JIRA and drill down. Thanks.
On Mon, Nov 23, 2015 at 4:18 PM, Anshum Gupta <an...@anshumgupta.net>
wrote:
> To restart solr, you should instead use something like:
> bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> or
> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>
> I've seen others report the same exception but never ran into this one
> myself. Let me try this out.
>
>
>
> On Mon, Nov 23, 2015 at 2:55 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
>> FWIW
>>
>> I am getting fairly consistent results that if I follow the SOLR-8326
>> procedure just up through the step of "solr-5.3.1/bin/solr start -e cloud
>> -z localhost:2181": if I then stop just one node (either "./solr stop -p
>> 7574" or "./solr stop -p 8983") and then restart that same node (using the
>> command suggested by "solr-5.3.1/bin/solr start -e cloud -z
>> localhost:2181"), then the solr.log for the stopped-and-restarted node gets
>> such stack traces as
>> ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3
>> x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException;
>> Error while trying to recover.:java.util.concurrent.ExecutionException:
>> org.apache.http.ParseException: Invalid content type:
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> Caused by: org.apache.http.ParseException: Invalid content type:
>> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> While the node which stayed up the whole time starts getting such stack
>> traces as
>> ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3
>> x:gettingstarted_shard2_replica2]
>> org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
>> java.lang.NumberFormatException: For input string: "r?"
>> at
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:589)
>> at java.lang.Long.parseLong(Long.java:631)
>> at
>> org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> In this case the string is just "r?", but usually it is a longer string
>> of control characters.
>>
>> If I shutdown _both_ nodes and restart _one_, and then allow it to be
>> "Waiting until we see more replicas up" until it recognizes itself as
>> leader, and _then_ restart the other node -- in this case it successfully
>> starts.
>>
>> Is there some necessary environment tweaking? The symptoms seem similar
>> whether I use the security.json from SOLR-8326 or the security.json from
>> the Wiki (with the comma repositioned).
>>
>>
>>
>> -----Original Message-----
>> From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> Sent: Friday, November 20, 2015 6:59 PM
>> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>>
>> Thanks
>>
>> It seems to work when there is no security.json, so perhaps there's some
>> typo in the initial version.
>>
>> I notice that the version you sent is different from the documentation at
>> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
>> in that the Wiki version has "permissions" before "user-role": I also
>> notice that (at least as of right this moment) the Wiki version has a comma
>> at the end of '"user-role":{"solr":"admin"},' even though it is at the end:
>> and I notice that the Wiki version seems to lack a comma between the
>> "permissions" section and the "user-role" section. I just now also noticed
>> that the version you sent has '"user-role":{"solr":["admin"]}' (with square
>> brackets) whereas the Wiki does not have square brackets.
>>
>> The placement of the comma definitely looks wrong in the Wiki at the
>> moment (though perhaps someone might correct the Wiki before too long).
>> Other than that, I don’t know whether the order and/or the square brackets
>> make a difference. I can try with different permutations.
>>
>> Thanks again
>>
>> P.S. for the record, the Wiki currently has
>> {
>> "authentication":{
>> "class":"solr.BasicAuthPlugin",
>> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>> },
>> "authorization":{
>> "class":"solr.RuleBasedAuthorizationPlugin",
>> "permissions":[{"name":"security-edit",
>> "role":"admin"}]
>> "user-role":{"solr":"admin"},
>> }}
>>
>> -----Original Message-----
>> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> Sent: Friday, November 20, 2015 6:18 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>>
>> This seems unrelated and more like a user error somewhere. Can you just
>> follow the steps, without any security settings i.e. not even uploading
>> security.json and see if you still see this? Sorry, but I don't have
>> access
>> to the code right now, I try and look at this later tonight.
>>
>> On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> craig.oakley@nih.gov> wrote:
>>
>> > Thank you for opening SOLR-8326
>> >
>> > As a side note, in the procedure you listed, even before adding the
>> > collection-admin-edit authorization, I'm already hitting trouble:
>> stopping
>> > and restarting a node results in the following
>> >
>> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Publishing state of core solr8326_shard2_replica1 as recovering, leader
>> is
>> > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I
>> am
>> > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
>> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController;
>> publishing
>> > state=recovering
>> > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Publishing state of core solr8326_shard1_replica1 as recovering, leader
>> is
>> > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I
>> am
>> > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
>> > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController;
>> publishing
>> > state=recovering
>> > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> Sending
>> > prep recovery command to http://{IP-address-redacted}:8983/solr;
>> > WaitForState:
>> >
>> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
>> > INFO - 2015-11-20 22:48:41.289; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
>> > WatchedEvent state:SyncConnected type:NodeDataChanged
>> > path:/collections/solr8326/state.json for collection solr8326 has
>> occurred
>> > - updating... (live nodes size: 2)
>> > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> Sending
>> > prep recovery command to http://{IP-address-redacted}:8983/solr;
>> > WaitForState:
>> >
>> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
>> > INFO - 2015-11-20 22:48:41.291; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
>> to
>> > ver 25
>> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
>> > while trying to recover.:java.util.concurrent.ExecutionException:
>> > org.apache.http.ParseException: Invalid content type:
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> > at
>> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> > Caused by: org.apache.http.ParseException: Invalid content type:
>> > at
>> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > INFO - 2015-11-20 22:48:41.298; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
>> > WatchedEvent state:SyncConnected type:NodeDataChanged
>> > path:/collections/solr8326/state.json for collection solr8326 has
>> occurred
>> > - updating... (live nodes size: 2)
>> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Recovery failed - trying again... (4)
>> > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
>> > 32.0 seconds before trying to recover again (5)
>> > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
>> > while trying to recover.:java.util.concurrent.ExecutionException:
>> > org.apache.http.ParseException: Invalid content type:
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> > at
>> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> > Caused by: org.apache.http.ParseException: Invalid content type:
>> > at
>> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Recovery failed - trying again... (4)
>> > INFO - 2015-11-20 22:48:41.318; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
>> to
>> > ver 26
>> > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
>> > 32.0 seconds before trying to recover again (5)
>> >
>> >
>> > I would not be surprised if this were to be some unrelated issue (the
>> > symptoms are quite different)
>> >
>> >
>> >
>> > Thanks again
>> >
>> >
>> > -----Original Message-----
>> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> > Sent: Friday, November 20, 2015 1:31 PM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>> >
>> > Collections API were available before November of 2014, if that is when
>> you
>> > took the class. However, it was only with Solr 5.0 (released in Feb
>> 2015)
>> > that the only supported mechanism to create a collection was restricted
>> to
>> > Collections API.
>> >
>> > Here are the list of steps that you'd need to run to see that things are
>> > fine for you without the read permission:
>> > * Untar and setup Solr, don't start it yet
>> > * Start clean zookeeper
>> > * Put the security.json in zk, without anything other than a
>> security-edit
>> > permission. Find the content of the file below. Upload it using your
>> own zk
>> > client or through the solr script:
>> > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost
>> localhost:2181
>> > -cmd putfile /security.json ~/security.json
>> >
>> > security.json:
>> >
>> >
>> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> >
>> >
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>> >
>> > * Start solr:
>> > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>> >
>> > You would need to key in a few things e.g. #nodes and ports, leave them
>> at
>> > the default values of 2 nodes and 8983/7574, unless you want to run
>> Solr on
>> > a different port. Then let it create a default collection to just make
>> sure
>> > that everything works fine.
>> >
>> > * Add the collection-admin-edit command:
>> > > curl --user solr:SolrRocks
>> > http://localhost:8983/solr/admin/authorization
>> > -H 'Content-type:application/json' -d '{"set-permission" :
>> > {"name":"collection-admin-edit", "role":"admin"}}'
>> >
>> > At this point, everything should be working fine. Restarting the nodes
>> > should also work fine. You can try 2 things at this point:
>> > 1. Create a new collection with 1 shard and 1 replica and then try
>> adding a
>> > replica, here's how:
>> > > curl --user solr:SolrRocks
>> >
>> >
>> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>> >
>> > > curl --user solr:SolrRocks
>> >
>> >
>> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>> >
>> > This should work fine.
>> >
>> > 2. After this, try restarting the solr cluster. Here's how you can do
>> so,
>> > assuming you didn't change any of the defaults and you are running zk on
>> > localhost:2181. If not, just change those values below:
>> > > bin/solr stop -all
>> >
>> > After this, check that Solr was actually stopped. I'd also suggest you
>> tail
>> > the logs on both nodes when they are coming up to see any errors, if
>> any.
>> > The logs would be here: example/cloud/node1/logs/solr.log
>> > and example/cloud/node2/logs/solr.log
>> >
>> > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z
>> localhost:2181
>> > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z
>> localhost:2181
>> >
>> > If you get to this checkpoint fine, try adding a read permission.
>> > Add a permission:
>> > > curl --user solr:SolrRocks
>> > http://localhost:8983/solr/admin/authorization
>> > -H 'Content-type:application/json' -d '{"set-permission" :
>> {"name":"read",
>> > "role":"read"}}'
>> >
>> > Add a user:
>> > > curl --user solr:SolrRocks
>> > http://localhost:8983/solr/admin/authentication
>> > -H 'Content-type:application/json' -d '{"set-user" :
>> > {"solrread":"solrRocks"}}'
>> >
>> > Assign a role to the user:
>> > >curl --user solr:SolrRocks
>> http://localhost:8983/solr/admin/authorization
>> > -H 'Content-type:application/json' -d '{"set-user-role" :
>> > {"solrread":["read"]}}'
>> >
>> > After this, you should start having issues with ADDREPLICA.
>> > Also, as you would at this point have a collection with a shard that
>> has a
>> > replication factor > 1 (remember the ADDREPLICA we did earlier), you
>> would
>> > have issues when you restart the cluster again using the steps I
>> mentioned
>> > above.
>> >
>> >
>> > Can you confirm this? I guess I'll just use this text to create a new
>> JIRA
>> > now.
>> >
>> >
>> > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> > craig.oakley@nih.gov> wrote:
>> >
>> > > Thank you again for the reply.
>> > >
>> > > Below is the Email I was about to send prior to your reply a moment
>> ago:
>> > > shall I try again without "read" in the security.json?
>> > >
>> > >
>> > >
>> > > The Collections API method was not discussed in the "Unleashed" class
>> at
>> > > the conference in DC in 2014 (probably because it was not yet
>> available),
>> > > so I was using the method I knew.
>> > >
>> > > I have now tried again using admin/collections?action=CREATE (using
>> > > different port numbers to avoid confusion from the failed previous
>> > > attempts: the previously created nodes had been shutdown and their
>> > > core.properties files renamed so as not to be discovered), but the
>> > results
>> > > are the same:
>> > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> Starting
>> > > Replication Recovery.
>> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
>> > > buffering updates.
>> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
>> > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
>> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> > Attempting
>> > > to replicate from http://
>> > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
>> > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
>> > while
>> > > trying to
>> > >
>> >
>> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> > > Error from server at http://
>> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
>> > > Expected mime type application/octet-stream but got text/html. <html>
>> > > <head>
>> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>> > > <title>Error 401 Unauthorized request, Response code: 401</title>
>> > > </head>
>> > > <body><h2>HTTP ERROR 401</h2>
>> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
>> > > <pre> Unauthorized request, Response code:
>> > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>> > >
>> > > </body>
>> > > </html>
>> > >
>> > > at
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
>> > > at
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
>> > > at
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
>> > > at
>> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
>> > > at
>> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
>> > > at
>> > >
>> >
>> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
>> > > at
>> > >
>> >
>> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
>> > > at
>> > >
>> >
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
>> > > at
>> > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> > >
>> > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
>> > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
>> > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> Recovery
>> > > failed - trying again... (2)
>> > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait
>> 8.0
>> > > seconds before trying to recover again (3)
>> > >
>> > >
>> > > Below is a list of the steps I took.
>> > >
>> > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
>> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile
>> /security.json
>> > > ~/solr/security151119a.json
>> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
>> > > ../../solr/configsets/basic_configs/conf -confname xmpl3
>> > > cd ../../../bin/
>> > > ./solr -c -p 4695 -d
>> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
>> > -z
>> > > localhost:4545/solr/xmpl3 -s
>> > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
>> > > ./solr -c -p 4685 -d
>> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
>> > -z
>> > > localhost:4545/solr/xmpl3 -s
>> > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
>> > > curl -u solr:SolrRocks '
>> > >
>> >
>> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
>> > > '
>> > > curl -u solr:SolrRocks '
>> > >
>> >
>> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
>> > > '
>> > >
>> > >
>> > >
>> > >
>> > > Can you provide a list of steps to take in an out-of-the-box directory
>> > > tree whereby ADDREPLICA _will_ work with security.json already in
>> place?
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> > > Sent: Thursday, November 19, 2015 3:44 PM
>> > > To: solr-user@lucene.apache.org
>> > > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>> > >
>> > > I'll try out what you did later in the day, as soon as I get time but
>> why
>> > > exactly are you creating cores manually? Seems like you manually
>> create a
>> > > core and the try to add a replica. Can you try using the Collections
>> API
>> > to
>> > > create a collection?
>> > >
>> > > Starting Solr 5.0, the only supported way to create a new collection
>> is
>> > via
>> > > the Collections API. Creating a core would lead to a collection
>> creation
>> > > but that's not really supported. It was just something that was done
>> when
>> > > there were no Collections API.
>> > >
>> > >
>> > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> > > craig.oakley@nih.gov> wrote:
>> > >
>> > > > I tried again with the following security.json, but the results were
>> > the
>> > > > same:
>> > > >
>> > > > {
>> > > > "authentication":{
>> > > > "class":"solr.BasicAuthPlugin",
>> > > > "credentials":{
>> > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
>> > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
>> > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
>> > > > "":{"v":9}},
>> > > > "authorization":{
>> > > > "class":"solr.RuleBasedAuthorizationPlugin",
>> > > > "user-role":{
>> > > > "solr":[
>> > > > "admin",
>> > > > "read",
>> > > > "xmpladmin",
>> > > > "xmplgen",
>> > > > "xmplsel"],
>> > > > "solruser":[
>> > > > "read",
>> > > > "xmplgen",
>> > > > "xmplsel"]},
>> > > > "permissions":[
>> > > > {
>> > > > "name":"security-edit",
>> > > > "role":"admin"},
>> > > > {
>> > > > "name":"xmpl_admin",
>> > > > "collection":"xmpl",
>> > > > "path":"/admin/*",
>> > > > "role":"xmpladmin"},
>> > > > {
>> > > > "name":"xmpl_sel",
>> > > > "collection":"xmpl",
>> > > > "path":"/select/*",
>> > > > "role":null},
>> > > > {
>> > > > "name":"all-admin",
>> > > > "collection":null,
>> > > > "path":"/*",
>> > > > "role":"xmplgen"},
>> > > > {
>> > > > "name":"all-core-handlers",
>> > > > "path":"/*",
>> > > > "role":"xmplgen"}],
>> > > > "":{"v":42}}}
>> > > >
>> > > > -----Original Message-----
>> > > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> > > > Sent: Thursday, November 19, 2015 1:46 PM
>> > > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> > > > Subject: RE: Re:Re: Implementing security.json is breaking
>> ADDREPLICA
>> > > >
>> > > > I note that the thread called "Security Problems" (most recent post
>> by
>> > > > Nobel Paul) seems like it may help with much of what I'm trying to
>> do.
>> > I
>> > > > will see to what extent that may help.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Anshum Gupta
>> > >
>> >
>> >
>> >
>> > --
>> > Anshum Gupta
>> >
>>
>>
>>
>> --
>> Anshum Gupta
>>
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
Yes, I see the same issue. I'll update the JIRA and drill down. Thanks.
On Mon, Nov 23, 2015 at 4:18 PM, Anshum Gupta <an...@anshumgupta.net>
wrote:
> To restart solr, you should instead use something like:
> bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> or
> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>
> I've seen others report the same exception but never ran into this one
> myself. Let me try this out.
>
>
>
> On Mon, Nov 23, 2015 at 2:55 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
>> FWIW
>>
>> I am getting fairly consistent results that if I follow the SOLR-8326
>> procedure just up through the step of "solr-5.3.1/bin/solr start -e cloud
>> -z localhost:2181": if I then stop just one node (either "./solr stop -p
>> 7574" or "./solr stop -p 8983") and then restart that same node (using the
>> command suggested by "solr-5.3.1/bin/solr start -e cloud -z
>> localhost:2181"), then the solr.log for the stopped-and-restarted node gets
>> such stack traces as
>> ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3
>> x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException;
>> Error while trying to recover.:java.util.concurrent.ExecutionException:
>> org.apache.http.ParseException: Invalid content type:
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> Caused by: org.apache.http.ParseException: Invalid content type:
>> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> While the node which stayed up the whole time starts getting such stack
>> traces as
>> ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3
>> x:gettingstarted_shard2_replica2]
>> org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
>> java.lang.NumberFormatException: For input string: "r?"
>> at
>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>> at java.lang.Long.parseLong(Long.java:589)
>> at java.lang.Long.parseLong(Long.java:631)
>> at
>> org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
>> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>> at
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> at org.eclipse.jetty.server.Server.handle(Server.java:499)
>> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> In this case the string is just "r?", but usually it is a longer string
>> of control characters.
>>
>> If I shutdown _both_ nodes and restart _one_, and then allow it to be
>> "Waiting until we see more replicas up" until it recognizes itself as
>> leader, and _then_ restart the other node -- in this case it successfully
>> starts.
>>
>> Is there some necessary environment tweaking? The symptoms seem similar
>> whether I use the security.json from SOLR-8326 or the security.json from
>> the Wiki (with the comma repositioned).
>>
>>
>>
>> -----Original Message-----
>> From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> Sent: Friday, November 20, 2015 6:59 PM
>> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>>
>> Thanks
>>
>> It seems to work when there is no security.json, so perhaps there's some
>> typo in the initial version.
>>
>> I notice that the version you sent is different from the documentation at
>> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
>> in that the Wiki version has "permissions" before "user-role": I also
>> notice that (at least as of right this moment) the Wiki version has a comma
>> at the end of '"user-role":{"solr":"admin"},' even though it is at the end:
>> and I notice that the Wiki version seems to lack a comma between the
>> "permissions" section and the "user-role" section. I just now also noticed
>> that the version you sent has '"user-role":{"solr":["admin"]}' (with square
>> brackets) whereas the Wiki does not have square brackets.
>>
>> The placement of the comma definitely looks wrong in the Wiki at the
>> moment (though perhaps someone might correct the Wiki before too long).
>> Other than that, I don’t know whether the order and/or the square brackets
>> make a difference. I can try with different permutations.
>>
>> Thanks again
>>
>> P.S. for the record, the Wiki currently has
>> {
>> "authentication":{
>> "class":"solr.BasicAuthPlugin",
>> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>> },
>> "authorization":{
>> "class":"solr.RuleBasedAuthorizationPlugin",
>> "permissions":[{"name":"security-edit",
>> "role":"admin"}]
>> "user-role":{"solr":"admin"},
>> }}
>>
>> -----Original Message-----
>> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> Sent: Friday, November 20, 2015 6:18 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>>
>> This seems unrelated and more like a user error somewhere. Can you just
>> follow the steps, without any security settings i.e. not even uploading
>> security.json and see if you still see this? Sorry, but I don't have
>> access
>> to the code right now, I try and look at this later tonight.
>>
>> On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> craig.oakley@nih.gov> wrote:
>>
>> > Thank you for opening SOLR-8326
>> >
>> > As a side note, in the procedure you listed, even before adding the
>> > collection-admin-edit authorization, I'm already hitting trouble:
>> stopping
>> > and restarting a node results in the following
>> >
>> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Publishing state of core solr8326_shard2_replica1 as recovering, leader
>> is
>> > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I
>> am
>> > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
>> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController;
>> publishing
>> > state=recovering
>> > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Publishing state of core solr8326_shard1_replica1 as recovering, leader
>> is
>> > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I
>> am
>> > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
>> > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController;
>> publishing
>> > state=recovering
>> > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> Sending
>> > prep recovery command to http://{IP-address-redacted}:8983/solr;
>> > WaitForState:
>> >
>> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
>> > INFO - 2015-11-20 22:48:41.289; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
>> > WatchedEvent state:SyncConnected type:NodeDataChanged
>> > path:/collections/solr8326/state.json for collection solr8326 has
>> occurred
>> > - updating... (live nodes size: 2)
>> > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> Sending
>> > prep recovery command to http://{IP-address-redacted}:8983/solr;
>> > WaitForState:
>> >
>> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
>> > INFO - 2015-11-20 22:48:41.291; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
>> to
>> > ver 25
>> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
>> > while trying to recover.:java.util.concurrent.ExecutionException:
>> > org.apache.http.ParseException: Invalid content type:
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> > at
>> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> > Caused by: org.apache.http.ParseException: Invalid content type:
>> > at
>> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > INFO - 2015-11-20 22:48:41.298; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
>> > WatchedEvent state:SyncConnected type:NodeDataChanged
>> > path:/collections/solr8326/state.json for collection solr8326 has
>> occurred
>> > - updating... (live nodes size: 2)
>> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Recovery failed - trying again... (4)
>> > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
>> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
>> > 32.0 seconds before trying to recover again (5)
>> > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
>> > while trying to recover.:java.util.concurrent.ExecutionException:
>> > org.apache.http.ParseException: Invalid content type:
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
>> > at
>> >
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
>> > at
>> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> > Caused by: org.apache.http.ParseException: Invalid content type:
>> > at
>> org.apache.http.entity.ContentType.parse(ContentType.java:273)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
>> > at
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
>> > Recovery failed - trying again... (4)
>> > INFO - 2015-11-20 22:48:41.318; [ ]
>> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326
>> to
>> > ver 26
>> > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
>> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
>> > 32.0 seconds before trying to recover again (5)
>> >
>> >
>> > I would not be surprised if this were to be some unrelated issue (the
>> > symptoms are quite different)
>> >
>> >
>> >
>> > Thanks again
>> >
>> >
>> > -----Original Message-----
>> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> > Sent: Friday, November 20, 2015 1:31 PM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>> >
>> > Collections API were available before November of 2014, if that is when
>> you
>> > took the class. However, it was only with Solr 5.0 (released in Feb
>> 2015)
>> > that the only supported mechanism to create a collection was restricted
>> to
>> > Collections API.
>> >
>> > Here are the list of steps that you'd need to run to see that things are
>> > fine for you without the read permission:
>> > * Untar and setup Solr, don't start it yet
>> > * Start clean zookeeper
>> > * Put the security.json in zk, without anything other than a
>> security-edit
>> > permission. Find the content of the file below. Upload it using your
>> own zk
>> > client or through the solr script:
>> > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost
>> localhost:2181
>> > -cmd putfile /security.json ~/security.json
>> >
>> > security.json:
>> >
>> >
>> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> >
>> >
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>> >
>> > * Start solr:
>> > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>> >
>> > You would need to key in a few things e.g. #nodes and ports, leave them
>> at
>> > the default values of 2 nodes and 8983/7574, unless you want to run
>> Solr on
>> > a different port. Then let it create a default collection to just make
>> sure
>> > that everything works fine.
>> >
>> > * Add the collection-admin-edit command:
>> > > curl --user solr:SolrRocks
>> > http://localhost:8983/solr/admin/authorization
>> > -H 'Content-type:application/json' -d '{"set-permission" :
>> > {"name":"collection-admin-edit", "role":"admin"}}'
>> >
>> > At this point, everything should be working fine. Restarting the nodes
>> > should also work fine. You can try 2 things at this point:
>> > 1. Create a new collection with 1 shard and 1 replica and then try
>> adding a
>> > replica, here's how:
>> > > curl --user solr:SolrRocks
>> >
>> >
>> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>> >
>> > > curl --user solr:SolrRocks
>> >
>> >
>> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>> >
>> > This should work fine.
>> >
>> > 2. After this, try restarting the solr cluster. Here's how you can do
>> so,
>> > assuming you didn't change any of the defaults and you are running zk on
>> > localhost:2181. If not, just change those values below:
>> > > bin/solr stop -all
>> >
>> > After this, check that Solr was actually stopped. I'd also suggest you
>> tail
>> > the logs on both nodes when they are coming up to see any errors, if
>> any.
>> > The logs would be here: example/cloud/node1/logs/solr.log
>> > and example/cloud/node2/logs/solr.log
>> >
>> > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z
>> localhost:2181
>> > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z
>> localhost:2181
>> >
>> > If you get to this checkpoint fine, try adding a read permission.
>> > Add a permission:
>> > > curl --user solr:SolrRocks
>> > http://localhost:8983/solr/admin/authorization
>> > -H 'Content-type:application/json' -d '{"set-permission" :
>> {"name":"read",
>> > "role":"read"}}'
>> >
>> > Add a user:
>> > > curl --user solr:SolrRocks
>> > http://localhost:8983/solr/admin/authentication
>> > -H 'Content-type:application/json' -d '{"set-user" :
>> > {"solrread":"solrRocks"}}'
>> >
>> > Assign a role to the user:
>> > >curl --user solr:SolrRocks
>> http://localhost:8983/solr/admin/authorization
>> > -H 'Content-type:application/json' -d '{"set-user-role" :
>> > {"solrread":["read"]}}'
>> >
>> > After this, you should start having issues with ADDREPLICA.
>> > Also, as you would at this point have a collection with a shard that
>> has a
>> > replication factor > 1 (remember the ADDREPLICA we did earlier), you
>> would
>> > have issues when you restart the cluster again using the steps I
>> mentioned
>> > above.
>> >
>> >
>> > Can you confirm this? I guess I'll just use this text to create a new
>> JIRA
>> > now.
>> >
>> >
>> > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> > craig.oakley@nih.gov> wrote:
>> >
>> > > Thank you again for the reply.
>> > >
>> > > Below is the Email I was about to send prior to your reply a moment
>> ago:
>> > > shall I try again without "read" in the security.json?
>> > >
>> > >
>> > >
>> > > The Collections API method was not discussed in the "Unleashed" class
>> at
>> > > the conference in DC in 2014 (probably because it was not yet
>> available),
>> > > so I was using the method I knew.
>> > >
>> > > I have now tried again using admin/collections?action=CREATE (using
>> > > different port numbers to avoid confusion from the failed previous
>> > > attempts: the previously created nodes had been shutdown and their
>> > > core.properties files renamed so as not to be discovered), but the
>> > results
>> > > are the same:
>> > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> Starting
>> > > Replication Recovery.
>> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
>> > > buffering updates.
>> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
>> > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
>> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> > Attempting
>> > > to replicate from http://
>> > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
>> > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
>> > while
>> > > trying to
>> > >
>> >
>> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> > > Error from server at http://
>> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
>> > > Expected mime type application/octet-stream but got text/html. <html>
>> > > <head>
>> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>> > > <title>Error 401 Unauthorized request, Response code: 401</title>
>> > > </head>
>> > > <body><h2>HTTP ERROR 401</h2>
>> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
>> > > <pre> Unauthorized request, Response code:
>> > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>> > >
>> > > </body>
>> > > </html>
>> > >
>> > > at
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
>> > > at
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
>> > > at
>> > >
>> >
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
>> > > at
>> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
>> > > at
>> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
>> > > at
>> > >
>> >
>> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
>> > > at
>> > >
>> >
>> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
>> > > at
>> > >
>> >
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
>> > > at
>> > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>> > >
>> > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
>> > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
>> > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
>> Recovery
>> > > failed - trying again... (2)
>> > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
>> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait
>> 8.0
>> > > seconds before trying to recover again (3)
>> > >
>> > >
>> > > Below is a list of the steps I took.
>> > >
>> > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
>> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile
>> /security.json
>> > > ~/solr/security151119a.json
>> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
>> > > ../../solr/configsets/basic_configs/conf -confname xmpl3
>> > > cd ../../../bin/
>> > > ./solr -c -p 4695 -d
>> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
>> > -z
>> > > localhost:4545/solr/xmpl3 -s
>> > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
>> > > ./solr -c -p 4685 -d
>> ~dbman/solr/straight531outofbox/solr-5.3.1/server/
>> > -z
>> > > localhost:4545/solr/xmpl3 -s
>> > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
>> > > curl -u solr:SolrRocks '
>> > >
>> >
>> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
>> > > '
>> > > curl -u solr:SolrRocks '
>> > >
>> >
>> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
>> > > '
>> > >
>> > >
>> > >
>> > >
>> > > Can you provide a list of steps to take in an out-of-the-box directory
>> > > tree whereby ADDREPLICA _will_ work with security.json already in
>> place?
>> > >
>> > >
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
>> > > Sent: Thursday, November 19, 2015 3:44 PM
>> > > To: solr-user@lucene.apache.org
>> > > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>> > >
>> > > I'll try out what you did later in the day, as soon as I get time but
>> why
>> > > exactly are you creating cores manually? Seems like you manually
>> create a
>> > > core and the try to add a replica. Can you try using the Collections
>> API
>> > to
>> > > create a collection?
>> > >
>> > > Starting Solr 5.0, the only supported way to create a new collection
>> is
>> > via
>> > > the Collections API. Creating a core would lead to a collection
>> creation
>> > > but that's not really supported. It was just something that was done
>> when
>> > > there were no Collections API.
>> > >
>> > >
>> > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> > > craig.oakley@nih.gov> wrote:
>> > >
>> > > > I tried again with the following security.json, but the results were
>> > the
>> > > > same:
>> > > >
>> > > > {
>> > > > "authentication":{
>> > > > "class":"solr.BasicAuthPlugin",
>> > > > "credentials":{
>> > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
>> > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
>> > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
>> > > > "":{"v":9}},
>> > > > "authorization":{
>> > > > "class":"solr.RuleBasedAuthorizationPlugin",
>> > > > "user-role":{
>> > > > "solr":[
>> > > > "admin",
>> > > > "read",
>> > > > "xmpladmin",
>> > > > "xmplgen",
>> > > > "xmplsel"],
>> > > > "solruser":[
>> > > > "read",
>> > > > "xmplgen",
>> > > > "xmplsel"]},
>> > > > "permissions":[
>> > > > {
>> > > > "name":"security-edit",
>> > > > "role":"admin"},
>> > > > {
>> > > > "name":"xmpl_admin",
>> > > > "collection":"xmpl",
>> > > > "path":"/admin/*",
>> > > > "role":"xmpladmin"},
>> > > > {
>> > > > "name":"xmpl_sel",
>> > > > "collection":"xmpl",
>> > > > "path":"/select/*",
>> > > > "role":null},
>> > > > {
>> > > > "name":"all-admin",
>> > > > "collection":null,
>> > > > "path":"/*",
>> > > > "role":"xmplgen"},
>> > > > {
>> > > > "name":"all-core-handlers",
>> > > > "path":"/*",
>> > > > "role":"xmplgen"}],
>> > > > "":{"v":42}}}
>> > > >
>> > > > -----Original Message-----
>> > > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
>> > > > Sent: Thursday, November 19, 2015 1:46 PM
>> > > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
>> > > > Subject: RE: Re:Re: Implementing security.json is breaking
>> ADDREPLICA
>> > > >
>> > > > I note that the thread called "Security Problems" (most recent post
>> by
>> > > > Nobel Paul) seems like it may help with much of what I'm trying to
>> do.
>> > I
>> > > > will see to what extent that may help.
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Anshum Gupta
>> > >
>> >
>> >
>> >
>> > --
>> > Anshum Gupta
>> >
>>
>>
>>
>> --
>> Anshum Gupta
>>
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
To restart solr, you should instead use something like:
bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
or
bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
I've seen others report the same exception but never ran into this one
myself. Let me try this out.
On Mon, Nov 23, 2015 at 2:55 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> FWIW
>
> I am getting fairly consistent results that if I follow the SOLR-8326
> procedure just up through the step of "solr-5.3.1/bin/solr start -e cloud
> -z localhost:2181": if I then stop just one node (either "./solr stop -p
> 7574" or "./solr stop -p 8983") and then restart that same node (using the
> command suggested by "solr-5.3.1/bin/solr start -e cloud -z
> localhost:2181"), then the solr.log for the stopped-and-restarted node gets
> such stack traces as
> ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3
> x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException;
> Error while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> While the node which stayed up the whole time starts getting such stack
> traces as
> ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3
> x:gettingstarted_shard2_replica2]
> org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
> java.lang.NumberFormatException: For input string: "r?"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589)
> at java.lang.Long.parseLong(Long.java:631)
> at
> org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
> at
> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
>
> In this case the string is just "r?", but usually it is a longer string of
> control characters.
>
> If I shutdown _both_ nodes and restart _one_, and then allow it to be
> "Waiting until we see more replicas up" until it recognizes itself as
> leader, and _then_ restart the other node -- in this case it successfully
> starts.
>
> Is there some necessary environment tweaking? The symptoms seem similar
> whether I use the security.json from SOLR-8326 or the security.json from
> the Wiki (with the comma repositioned).
>
>
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Friday, November 20, 2015 6:59 PM
> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Thanks
>
> It seems to work when there is no security.json, so perhaps there's some
> typo in the initial version.
>
> I notice that the version you sent is different from the documentation at
> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
> in that the Wiki version has "permissions" before "user-role": I also
> notice that (at least as of right this moment) the Wiki version has a comma
> at the end of '"user-role":{"solr":"admin"},' even though it is at the end:
> and I notice that the Wiki version seems to lack a comma between the
> "permissions" section and the "user-role" section. I just now also noticed
> that the version you sent has '"user-role":{"solr":["admin"]}' (with square
> brackets) whereas the Wiki does not have square brackets.
>
> The placement of the comma definitely looks wrong in the Wiki at the
> moment (though perhaps someone might correct the Wiki before too long).
> Other than that, I don’t know whether the order and/or the square brackets
> make a difference. I can try with different permutations.
>
> Thanks again
>
> P.S. for the record, the Wiki currently has
> {
> "authentication":{
> "class":"solr.BasicAuthPlugin",
> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
> },
> "authorization":{
> "class":"solr.RuleBasedAuthorizationPlugin",
> "permissions":[{"name":"security-edit",
> "role":"admin"}]
> "user-role":{"solr":"admin"},
> }}
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Friday, November 20, 2015 6:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> This seems unrelated and more like a user error somewhere. Can you just
> follow the steps, without any security settings i.e. not even uploading
> security.json and see if you still see this? Sorry, but I don't have access
> to the code right now, I try and look at this later tonight.
>
> On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > Thank you for opening SOLR-8326
> >
> > As a side note, in the procedure you listed, even before adding the
> > collection-admin-edit authorization, I'm already hitting trouble:
> stopping
> > and restarting a node results in the following
> >
> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> > Publishing state of core solr8326_shard2_replica1 as recovering, leader
> is
> > http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I
> am
> > http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> > INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> > x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController;
> publishing
> > state=recovering
> > INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> > Publishing state of core solr8326_shard1_replica1 as recovering, leader
> is
> > http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I
> am
> > http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> > INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> > x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController;
> publishing
> > state=recovering
> > INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Sending
> > prep recovery command to http://{IP-address-redacted}:8983/solr;
> > WaitForState:
> >
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> > INFO - 2015-11-20 22:48:41.289; [ ]
> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> > WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/solr8326/state.json for collection solr8326 has
> occurred
> > - updating... (live nodes size: 2)
> > INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Sending
> > prep recovery command to http://{IP-address-redacted}:8983/solr;
> > WaitForState:
> >
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> > INFO - 2015-11-20 22:48:41.291; [ ]
> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> > ver 25
> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> > x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
> > while trying to recover.:java.util.concurrent.ExecutionException:
> > org.apache.http.ParseException: Invalid content type:
> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> > Caused by: org.apache.http.ParseException: Invalid content type:
> > at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > INFO - 2015-11-20 22:48:41.298; [ ]
> > org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> > WatchedEvent state:SyncConnected type:NodeDataChanged
> > path:/collections/solr8326/state.json for collection solr8326 has
> occurred
> > - updating... (live nodes size: 2)
> > ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> > Recovery failed - trying again... (4)
> > INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> > x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> > 32.0 seconds before trying to recover again (5)
> > ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> > x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
> > while trying to recover.:java.util.concurrent.ExecutionException:
> > org.apache.http.ParseException: Invalid content type:
> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> > Caused by: org.apache.http.ParseException: Invalid content type:
> > at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> > Recovery failed - trying again... (4)
> > INFO - 2015-11-20 22:48:41.318; [ ]
> > org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> > ver 26
> > INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> > x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> > 32.0 seconds before trying to recover again (5)
> >
> >
> > I would not be surprised if this were to be some unrelated issue (the
> > symptoms are quite different)
> >
> >
> >
> > Thanks again
> >
> >
> > -----Original Message-----
> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> > Sent: Friday, November 20, 2015 1:31 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > Collections API were available before November of 2014, if that is when
> you
> > took the class. However, it was only with Solr 5.0 (released in Feb 2015)
> > that the only supported mechanism to create a collection was restricted
> to
> > Collections API.
> >
> > Here are the list of steps that you'd need to run to see that things are
> > fine for you without the read permission:
> > * Untar and setup Solr, don't start it yet
> > * Start clean zookeeper
> > * Put the security.json in zk, without anything other than a
> security-edit
> > permission. Find the content of the file below. Upload it using your own
> zk
> > client or through the solr script:
> > > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> > -cmd putfile /security.json ~/security.json
> >
> > security.json:
> >
> >
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> >
> >
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
> >
> > * Start solr:
> > > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
> >
> > You would need to key in a few things e.g. #nodes and ports, leave them
> at
> > the default values of 2 nodes and 8983/7574, unless you want to run Solr
> on
> > a different port. Then let it create a default collection to just make
> sure
> > that everything works fine.
> >
> > * Add the collection-admin-edit command:
> > > curl --user solr:SolrRocks
> > http://localhost:8983/solr/admin/authorization
> > -H 'Content-type:application/json' -d '{"set-permission" :
> > {"name":"collection-admin-edit", "role":"admin"}}'
> >
> > At this point, everything should be working fine. Restarting the nodes
> > should also work fine. You can try 2 things at this point:
> > 1. Create a new collection with 1 shard and 1 replica and then try
> adding a
> > replica, here's how:
> > > curl --user solr:SolrRocks
> >
> >
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
> >
> > > curl --user solr:SolrRocks
> >
> >
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
> >
> > This should work fine.
> >
> > 2. After this, try restarting the solr cluster. Here's how you can do so,
> > assuming you didn't change any of the defaults and you are running zk on
> > localhost:2181. If not, just change those values below:
> > > bin/solr stop -all
> >
> > After this, check that Solr was actually stopped. I'd also suggest you
> tail
> > the logs on both nodes when they are coming up to see any errors, if any.
> > The logs would be here: example/cloud/node1/logs/solr.log
> > and example/cloud/node2/logs/solr.log
> >
> > > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z
> localhost:2181
> > > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z
> localhost:2181
> >
> > If you get to this checkpoint fine, try adding a read permission.
> > Add a permission:
> > > curl --user solr:SolrRocks
> > http://localhost:8983/solr/admin/authorization
> > -H 'Content-type:application/json' -d '{"set-permission" :
> {"name":"read",
> > "role":"read"}}'
> >
> > Add a user:
> > > curl --user solr:SolrRocks
> > http://localhost:8983/solr/admin/authentication
> > -H 'Content-type:application/json' -d '{"set-user" :
> > {"solrread":"solrRocks"}}'
> >
> > Assign a role to the user:
> > >curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> > -H 'Content-type:application/json' -d '{"set-user-role" :
> > {"solrread":["read"]}}'
> >
> > After this, you should start having issues with ADDREPLICA.
> > Also, as you would at this point have a collection with a shard that has
> a
> > replication factor > 1 (remember the ADDREPLICA we did earlier), you
> would
> > have issues when you restart the cluster again using the steps I
> mentioned
> > above.
> >
> >
> > Can you confirm this? I guess I'll just use this text to create a new
> JIRA
> > now.
> >
> >
> > On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> > > Thank you again for the reply.
> > >
> > > Below is the Email I was about to send prior to your reply a moment
> ago:
> > > shall I try again without "read" in the security.json?
> > >
> > >
> > >
> > > The Collections API method was not discussed in the "Unleashed" class
> at
> > > the conference in DC in 2014 (probably because it was not yet
> available),
> > > so I was using the method I knew.
> > >
> > > I have now tried again using admin/collections?action=CREATE (using
> > > different port numbers to avoid confusion from the failed previous
> > > attempts: the previously created nodes had been shutdown and their
> > > core.properties files renamed so as not to be discovered), but the
> > results
> > > are the same:
> > > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Starting
> > > Replication Recovery.
> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> > > buffering updates.
> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> > > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> > > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> > Attempting
> > > to replicate from http://
> > > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> > > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> > while
> > > trying to
> > >
> >
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > > Error from server at http://
> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> > > Expected mime type application/octet-stream but got text/html. <html>
> > > <head>
> > > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> > > <title>Error 401 Unauthorized request, Response code: 401</title>
> > > </head>
> > > <body><h2>HTTP ERROR 401</h2>
> > > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> > > <pre> Unauthorized request, Response code:
> > > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> > >
> > > </body>
> > > </html>
> > >
> > > at
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> > > at
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> > > at
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> > > at
> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> > > at
> > > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> > > at
> > >
> >
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> > > at
> > >
> >
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> > > at
> > >
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> > > at
> > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> > >
> > > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> > > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> > > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Recovery
> > > failed - trying again... (2)
> > > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait
> 8.0
> > > seconds before trying to recover again (3)
> > >
> > >
> > > Below is a list of the steps I took.
> > >
> > > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile
> /security.json
> > > ~/solr/security151119a.json
> > > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> > > ../../solr/configsets/basic_configs/conf -confname xmpl3
> > > cd ../../../bin/
> > > ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> > -z
> > > localhost:4545/solr/xmpl3 -s
> > > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> > > ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> > -z
> > > localhost:4545/solr/xmpl3 -s
> > > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> > > curl -u solr:SolrRocks '
> > >
> >
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> > > '
> > > curl -u solr:SolrRocks '
> > >
> >
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> > > '
> > >
> > >
> > >
> > >
> > > Can you provide a list of steps to take in an out-of-the-box directory
> > > tree whereby ADDREPLICA _will_ work with security.json already in
> place?
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> > > Sent: Thursday, November 19, 2015 3:44 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> > >
> > > I'll try out what you did later in the day, as soon as I get time but
> why
> > > exactly are you creating cores manually? Seems like you manually
> create a
> > > core and the try to add a replica. Can you try using the Collections
> API
> > to
> > > create a collection?
> > >
> > > Starting Solr 5.0, the only supported way to create a new collection is
> > via
> > > the Collections API. Creating a core would lead to a collection
> creation
> > > but that's not really supported. It was just something that was done
> when
> > > there were no Collections API.
> > >
> > >
> > > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > > craig.oakley@nih.gov> wrote:
> > >
> > > > I tried again with the following security.json, but the results were
> > the
> > > > same:
> > > >
> > > > {
> > > > "authentication":{
> > > > "class":"solr.BasicAuthPlugin",
> > > > "credentials":{
> > > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > > > "":{"v":9}},
> > > > "authorization":{
> > > > "class":"solr.RuleBasedAuthorizationPlugin",
> > > > "user-role":{
> > > > "solr":[
> > > > "admin",
> > > > "read",
> > > > "xmpladmin",
> > > > "xmplgen",
> > > > "xmplsel"],
> > > > "solruser":[
> > > > "read",
> > > > "xmplgen",
> > > > "xmplsel"]},
> > > > "permissions":[
> > > > {
> > > > "name":"security-edit",
> > > > "role":"admin"},
> > > > {
> > > > "name":"xmpl_admin",
> > > > "collection":"xmpl",
> > > > "path":"/admin/*",
> > > > "role":"xmpladmin"},
> > > > {
> > > > "name":"xmpl_sel",
> > > > "collection":"xmpl",
> > > > "path":"/select/*",
> > > > "role":null},
> > > > {
> > > > "name":"all-admin",
> > > > "collection":null,
> > > > "path":"/*",
> > > > "role":"xmplgen"},
> > > > {
> > > > "name":"all-core-handlers",
> > > > "path":"/*",
> > > > "role":"xmplgen"}],
> > > > "":{"v":42}}}
> > > >
> > > > -----Original Message-----
> > > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > > > Sent: Thursday, November 19, 2015 1:46 PM
> > > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> > > > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> > > >
> > > > I note that the thread called "Security Problems" (most recent post
> by
> > > > Nobel Paul) seems like it may help with much of what I'm trying to
> do.
> > I
> > > > will see to what extent that may help.
> > > >
> > >
> > >
> > >
> > > --
> > > Anshum Gupta
> > >
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
FWIW
I am getting fairly consistent results that if I follow the SOLR-8326 procedure just up through the step of "solr-5.3.1/bin/solr start -e cloud -z localhost:2181": if I then stop just one node (either "./solr stop -p 7574" or "./solr stop -p 8983") and then restart that same node (using the command suggested by "solr-5.3.1/bin/solr start -e cloud -z localhost:2181"), then the solr.log for the stopped-and-restarted node gets such stack traces as
ERROR - 2015-11-23 21:49:28.663; [c:gettingstarted s:shard2 r:core_node3 x:gettingstarted_shard2_replica2] org.apache.solr.common.SolrException; Error while trying to recover.:java.util.concurrent.ExecutionException: org.apache.http.ParseException: Invalid content type:
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Caused by: org.apache.http.ParseException: Invalid content type:
at org.apache.http.entity.ContentType.parse(ContentType.java:273)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
While the node which stayed up the whole time starts getting such stack traces as
ERROR - 2015-11-23 21:57:46.019; [c:gettingstarted s:shard2 r:core_node3 x:gettingstarted_shard2_replica2] org.apache.solr.security.PKIAuthenticationPlugin; Invalid time r?
java.lang.NumberFormatException: For input string: "r?"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:128)
at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:252)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
In this case the string is just "r?", but usually it is a longer string of control characters.
If I shutdown _both_ nodes and restart _one_, and then allow it to be "Waiting until we see more replicas up" until it recognizes itself as leader, and _then_ restart the other node -- in this case it successfully starts.
Is there some necessary environment tweaking? The symptoms seem similar whether I use the security.json from SOLR-8326 or the security.json from the Wiki (with the comma repositioned).
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Friday, November 20, 2015 6:59 PM
To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Thanks
It seems to work when there is no security.json, so perhaps there's some typo in the initial version.
I notice that the version you sent is different from the documentation at cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins in that the Wiki version has "permissions" before "user-role": I also notice that (at least as of right this moment) the Wiki version has a comma at the end of '"user-role":{"solr":"admin"},' even though it is at the end: and I notice that the Wiki version seems to lack a comma between the "permissions" section and the "user-role" section. I just now also noticed that the version you sent has '"user-role":{"solr":["admin"]}' (with square brackets) whereas the Wiki does not have square brackets.
The placement of the comma definitely looks wrong in the Wiki at the moment (though perhaps someone might correct the Wiki before too long). Other than that, I don’t know whether the order and/or the square brackets make a difference. I can try with different permutations.
Thanks again
P.S. for the record, the Wiki currently has
{
"authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[{"name":"security-edit",
"role":"admin"}]
"user-role":{"solr":"admin"},
}}
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Friday, November 20, 2015 6:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
This seems unrelated and more like a user error somewhere. Can you just
follow the steps, without any security settings i.e. not even uploading
security.json and see if you still see this? Sorry, but I don't have access
to the code right now, I try and look at this later tonight.
On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you for opening SOLR-8326
>
> As a side note, in the procedure you listed, even before adding the
> collection-admin-edit authorization, I'm already hitting trouble: stopping
> and restarting a node results in the following
>
> INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard2_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard1_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO - 2015-11-20 22:48:41.289; [ ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO - 2015-11-20 22:48:41.291; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 25
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> INFO - 2015-11-20 22:48:41.298; [ ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
> ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO - 2015-11-20 22:48:41.318; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 26
> INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
>
>
> I would not be surprised if this were to be some unrelated issue (the
> symptoms are quite different)
>
>
>
> Thanks again
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Friday, November 20, 2015 1:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Collections API were available before November of 2014, if that is when you
> took the class. However, it was only with Solr 5.0 (released in Feb 2015)
> that the only supported mechanism to create a collection was restricted to
> Collections API.
>
> Here are the list of steps that you'd need to run to see that things are
> fine for you without the read permission:
> * Untar and setup Solr, don't start it yet
> * Start clean zookeeper
> * Put the security.json in zk, without anything other than a security-edit
> permission. Find the content of the file below. Upload it using your own zk
> client or through the solr script:
> > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd putfile /security.json ~/security.json
>
> security.json:
>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>
> * Start solr:
> > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>
> You would need to key in a few things e.g. #nodes and ports, leave them at
> the default values of 2 nodes and 8983/7574, unless you want to run Solr on
> a different port. Then let it create a default collection to just make sure
> that everything works fine.
>
> * Add the collection-admin-edit command:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" :
> {"name":"collection-admin-edit", "role":"admin"}}'
>
> At this point, everything should be working fine. Restarting the nodes
> should also work fine. You can try 2 things at this point:
> 1. Create a new collection with 1 shard and 1 replica and then try adding a
> replica, here's how:
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>
> This should work fine.
>
> 2. After this, try restarting the solr cluster. Here's how you can do so,
> assuming you didn't change any of the defaults and you are running zk on
> localhost:2181. If not, just change those values below:
> > bin/solr stop -all
>
> After this, check that Solr was actually stopped. I'd also suggest you tail
> the logs on both nodes when they are coming up to see any errors, if any.
> The logs would be here: example/cloud/node1/logs/solr.log
> and example/cloud/node2/logs/solr.log
>
> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>
> If you get to this checkpoint fine, try adding a read permission.
> Add a permission:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read",
> "role":"read"}}'
>
> Add a user:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authentication
> -H 'Content-type:application/json' -d '{"set-user" :
> {"solrread":"solrRocks"}}'
>
> Assign a role to the user:
> >curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-user-role" :
> {"solrread":["read"]}}'
>
> After this, you should start having issues with ADDREPLICA.
> Also, as you would at this point have a collection with a shard that has a
> replication factor > 1 (remember the ADDREPLICA we did earlier), you would
> have issues when you restart the cluster again using the steps I mentioned
> above.
>
>
> Can you confirm this? I guess I'll just use this text to create a new JIRA
> now.
>
>
> On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > Thank you again for the reply.
> >
> > Below is the Email I was about to send prior to your reply a moment ago:
> > shall I try again without "read" in the security.json?
> >
> >
> >
> > The Collections API method was not discussed in the "Unleashed" class at
> > the conference in DC in 2014 (probably because it was not yet available),
> > so I was using the method I knew.
> >
> > I have now tried again using admin/collections?action=CREATE (using
> > different port numbers to avoid confusion from the failed previous
> > attempts: the previously created nodes had been shutdown and their
> > core.properties files renamed so as not to be discovered), but the
> results
> > are the same:
> > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting
> > Replication Recovery.
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> > buffering updates.
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Attempting
> > to replicate from http://
> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> while
> > trying to
> >
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at http://
> {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> > Expected mime type application/octet-stream but got text/html. <html>
> > <head>
> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> > <title>Error 401 Unauthorized request, Response code: 401</title>
> > </head>
> > <body><h2>HTTP ERROR 401</h2>
> > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> > <pre> Unauthorized request, Response code:
> > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> >
> > </body>
> > </html>
> >
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >
> > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery
> > failed - trying again... (2)
> > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> > seconds before trying to recover again (3)
> >
> >
> > Below is a list of the steps I took.
> >
> > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json
> > ~/solr/security151119a.json
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> > ../../solr/configsets/basic_configs/conf -confname xmpl3
> > cd ../../../bin/
> > ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> > ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> > '
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> > '
> >
> >
> >
> >
> > Can you provide a list of steps to take in an out-of-the-box directory
> > tree whereby ADDREPLICA _will_ work with security.json already in place?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> > Sent: Thursday, November 19, 2015 3:44 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > I'll try out what you did later in the day, as soon as I get time but why
> > exactly are you creating cores manually? Seems like you manually create a
> > core and the try to add a replica. Can you try using the Collections API
> to
> > create a collection?
> >
> > Starting Solr 5.0, the only supported way to create a new collection is
> via
> > the Collections API. Creating a core would lead to a collection creation
> > but that's not really supported. It was just something that was done when
> > there were no Collections API.
> >
> >
> > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> > > I tried again with the following security.json, but the results were
> the
> > > same:
> > >
> > > {
> > > "authentication":{
> > > "class":"solr.BasicAuthPlugin",
> > > "credentials":{
> > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > > "":{"v":9}},
> > > "authorization":{
> > > "class":"solr.RuleBasedAuthorizationPlugin",
> > > "user-role":{
> > > "solr":[
> > > "admin",
> > > "read",
> > > "xmpladmin",
> > > "xmplgen",
> > > "xmplsel"],
> > > "solruser":[
> > > "read",
> > > "xmplgen",
> > > "xmplsel"]},
> > > "permissions":[
> > > {
> > > "name":"security-edit",
> > > "role":"admin"},
> > > {
> > > "name":"xmpl_admin",
> > > "collection":"xmpl",
> > > "path":"/admin/*",
> > > "role":"xmpladmin"},
> > > {
> > > "name":"xmpl_sel",
> > > "collection":"xmpl",
> > > "path":"/select/*",
> > > "role":null},
> > > {
> > > "name":"all-admin",
> > > "collection":null,
> > > "path":"/*",
> > > "role":"xmplgen"},
> > > {
> > > "name":"all-core-handlers",
> > > "path":"/*",
> > > "role":"xmplgen"}],
> > > "":{"v":42}}}
> > >
> > > -----Original Message-----
> > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > > Sent: Thursday, November 19, 2015 1:46 PM
> > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> > > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> > >
> > > I note that the thread called "Security Problems" (most recent post by
> > > Nobel Paul) seems like it may help with much of what I'm trying to do.
> I
> > > will see to what extent that may help.
> > >
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thanks
It seems to work when there is no security.json, so perhaps there's some typo in the initial version.
I notice that the version you sent is different from the documentation at cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins in that the Wiki version has "permissions" before "user-role": I also notice that (at least as of right this moment) the Wiki version has a comma at the end of '"user-role":{"solr":"admin"},' even though it is at the end: and I notice that the Wiki version seems to lack a comma between the "permissions" section and the "user-role" section. I just now also noticed that the version you sent has '"user-role":{"solr":["admin"]}' (with square brackets) whereas the Wiki does not have square brackets.
The placement of the comma definitely looks wrong in the Wiki at the moment (though perhaps someone might correct the Wiki before too long). Other than that, I don’t know whether the order and/or the square brackets make a difference. I can try with different permutations.
Thanks again
P.S. for the record, the Wiki currently has
{
"authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[{"name":"security-edit",
"role":"admin"}]
"user-role":{"solr":"admin"},
}}
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Friday, November 20, 2015 6:18 PM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
This seems unrelated and more like a user error somewhere. Can you just
follow the steps, without any security settings i.e. not even uploading
security.json and see if you still see this? Sorry, but I don't have access
to the code right now, I try and look at this later tonight.
On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you for opening SOLR-8326
>
> As a side note, in the procedure you listed, even before adding the
> collection-admin-edit authorization, I'm already hitting trouble: stopping
> and restarting a node results in the following
>
> INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard2_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard1_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO - 2015-11-20 22:48:41.289; [ ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO - 2015-11-20 22:48:41.291; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 25
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> INFO - 2015-11-20 22:48:41.298; [ ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
> ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO - 2015-11-20 22:48:41.318; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 26
> INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
>
>
> I would not be surprised if this were to be some unrelated issue (the
> symptoms are quite different)
>
>
>
> Thanks again
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Friday, November 20, 2015 1:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Collections API were available before November of 2014, if that is when you
> took the class. However, it was only with Solr 5.0 (released in Feb 2015)
> that the only supported mechanism to create a collection was restricted to
> Collections API.
>
> Here are the list of steps that you'd need to run to see that things are
> fine for you without the read permission:
> * Untar and setup Solr, don't start it yet
> * Start clean zookeeper
> * Put the security.json in zk, without anything other than a security-edit
> permission. Find the content of the file below. Upload it using your own zk
> client or through the solr script:
> > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd putfile /security.json ~/security.json
>
> security.json:
>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>
> * Start solr:
> > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>
> You would need to key in a few things e.g. #nodes and ports, leave them at
> the default values of 2 nodes and 8983/7574, unless you want to run Solr on
> a different port. Then let it create a default collection to just make sure
> that everything works fine.
>
> * Add the collection-admin-edit command:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" :
> {"name":"collection-admin-edit", "role":"admin"}}'
>
> At this point, everything should be working fine. Restarting the nodes
> should also work fine. You can try 2 things at this point:
> 1. Create a new collection with 1 shard and 1 replica and then try adding a
> replica, here's how:
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>
> This should work fine.
>
> 2. After this, try restarting the solr cluster. Here's how you can do so,
> assuming you didn't change any of the defaults and you are running zk on
> localhost:2181. If not, just change those values below:
> > bin/solr stop -all
>
> After this, check that Solr was actually stopped. I'd also suggest you tail
> the logs on both nodes when they are coming up to see any errors, if any.
> The logs would be here: example/cloud/node1/logs/solr.log
> and example/cloud/node2/logs/solr.log
>
> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>
> If you get to this checkpoint fine, try adding a read permission.
> Add a permission:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read",
> "role":"read"}}'
>
> Add a user:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authentication
> -H 'Content-type:application/json' -d '{"set-user" :
> {"solrread":"solrRocks"}}'
>
> Assign a role to the user:
> >curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-user-role" :
> {"solrread":["read"]}}'
>
> After this, you should start having issues with ADDREPLICA.
> Also, as you would at this point have a collection with a shard that has a
> replication factor > 1 (remember the ADDREPLICA we did earlier), you would
> have issues when you restart the cluster again using the steps I mentioned
> above.
>
>
> Can you confirm this? I guess I'll just use this text to create a new JIRA
> now.
>
>
> On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > Thank you again for the reply.
> >
> > Below is the Email I was about to send prior to your reply a moment ago:
> > shall I try again without "read" in the security.json?
> >
> >
> >
> > The Collections API method was not discussed in the "Unleashed" class at
> > the conference in DC in 2014 (probably because it was not yet available),
> > so I was using the method I knew.
> >
> > I have now tried again using admin/collections?action=CREATE (using
> > different port numbers to avoid confusion from the failed previous
> > attempts: the previously created nodes had been shutdown and their
> > core.properties files renamed so as not to be discovered), but the
> results
> > are the same:
> > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting
> > Replication Recovery.
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> > buffering updates.
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Attempting
> > to replicate from http://
> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> while
> > trying to
> >
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at http://
> {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> > Expected mime type application/octet-stream but got text/html. <html>
> > <head>
> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> > <title>Error 401 Unauthorized request, Response code: 401</title>
> > </head>
> > <body><h2>HTTP ERROR 401</h2>
> > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> > <pre> Unauthorized request, Response code:
> > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> >
> > </body>
> > </html>
> >
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >
> > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery
> > failed - trying again... (2)
> > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> > seconds before trying to recover again (3)
> >
> >
> > Below is a list of the steps I took.
> >
> > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json
> > ~/solr/security151119a.json
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> > ../../solr/configsets/basic_configs/conf -confname xmpl3
> > cd ../../../bin/
> > ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> > ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> > '
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> > '
> >
> >
> >
> >
> > Can you provide a list of steps to take in an out-of-the-box directory
> > tree whereby ADDREPLICA _will_ work with security.json already in place?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> > Sent: Thursday, November 19, 2015 3:44 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > I'll try out what you did later in the day, as soon as I get time but why
> > exactly are you creating cores manually? Seems like you manually create a
> > core and the try to add a replica. Can you try using the Collections API
> to
> > create a collection?
> >
> > Starting Solr 5.0, the only supported way to create a new collection is
> via
> > the Collections API. Creating a core would lead to a collection creation
> > but that's not really supported. It was just something that was done when
> > there were no Collections API.
> >
> >
> > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> > > I tried again with the following security.json, but the results were
> the
> > > same:
> > >
> > > {
> > > "authentication":{
> > > "class":"solr.BasicAuthPlugin",
> > > "credentials":{
> > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > > "":{"v":9}},
> > > "authorization":{
> > > "class":"solr.RuleBasedAuthorizationPlugin",
> > > "user-role":{
> > > "solr":[
> > > "admin",
> > > "read",
> > > "xmpladmin",
> > > "xmplgen",
> > > "xmplsel"],
> > > "solruser":[
> > > "read",
> > > "xmplgen",
> > > "xmplsel"]},
> > > "permissions":[
> > > {
> > > "name":"security-edit",
> > > "role":"admin"},
> > > {
> > > "name":"xmpl_admin",
> > > "collection":"xmpl",
> > > "path":"/admin/*",
> > > "role":"xmpladmin"},
> > > {
> > > "name":"xmpl_sel",
> > > "collection":"xmpl",
> > > "path":"/select/*",
> > > "role":null},
> > > {
> > > "name":"all-admin",
> > > "collection":null,
> > > "path":"/*",
> > > "role":"xmplgen"},
> > > {
> > > "name":"all-core-handlers",
> > > "path":"/*",
> > > "role":"xmplgen"}],
> > > "":{"v":42}}}
> > >
> > > -----Original Message-----
> > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > > Sent: Thursday, November 19, 2015 1:46 PM
> > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> > > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> > >
> > > I note that the thread called "Security Problems" (most recent post by
> > > Nobel Paul) seems like it may help with much of what I'm trying to do.
> I
> > > will see to what extent that may help.
> > >
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
This seems unrelated and more like a user error somewhere. Can you just
follow the steps, without any security settings i.e. not even uploading
security.json and see if you still see this? Sorry, but I don't have access
to the code right now, I try and look at this later tonight.
On Fri, Nov 20, 2015 at 3:07 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you for opening SOLR-8326
>
> As a side note, in the procedure you listed, even before adding the
> collection-admin-edit authorization, I'm already hitting trouble: stopping
> and restarting a node results in the following
>
> INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard2_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
> INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Publishing state of core solr8326_shard1_replica1 as recovering, leader is
> http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I am
> http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
> INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController; publishing
> state=recovering
> INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO - 2015-11-20 22:48:41.289; [ ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending
> prep recovery command to http://{IP-address-redacted}:8983/solr;
> WaitForState:
> action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
> INFO - 2015-11-20 22:48:41.291; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 25
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> INFO - 2015-11-20 22:48:41.298; [ ]
> org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/solr8326/state.json for collection solr8326 has occurred
> - updating... (live nodes size: 2)
> ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4
> x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
> ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error
> while trying to recover.:java.util.concurrent.ExecutionException:
> org.apache.http.ParseException: Invalid content type:
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> Caused by: org.apache.http.ParseException: Invalid content type:
> at org.apache.http.entity.ContentType.parse(ContentType.java:273)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
> Recovery failed - trying again... (4)
> INFO - 2015-11-20 22:48:41.318; [ ]
> org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to
> ver 26
> INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3
> x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait
> 32.0 seconds before trying to recover again (5)
>
>
> I would not be surprised if this were to be some unrelated issue (the
> symptoms are quite different)
>
>
>
> Thanks again
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Friday, November 20, 2015 1:31 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> Collections API were available before November of 2014, if that is when you
> took the class. However, it was only with Solr 5.0 (released in Feb 2015)
> that the only supported mechanism to create a collection was restricted to
> Collections API.
>
> Here are the list of steps that you'd need to run to see that things are
> fine for you without the read permission:
> * Untar and setup Solr, don't start it yet
> * Start clean zookeeper
> * Put the security.json in zk, without anything other than a security-edit
> permission. Find the content of the file below. Upload it using your own zk
> client or through the solr script:
> > solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
> -cmd putfile /security.json ~/security.json
>
> security.json:
>
> {"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
>
> * Start solr:
> > solr-5.3.1/bin/solr start -e cloud -z localhost:2181
>
> You would need to key in a few things e.g. #nodes and ports, leave them at
> the default values of 2 nodes and 8983/7574, unless you want to run Solr on
> a different port. Then let it create a default collection to just make sure
> that everything works fine.
>
> * Add the collection-admin-edit command:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" :
> {"name":"collection-admin-edit", "role":"admin"}}'
>
> At this point, everything should be working fine. Restarting the nodes
> should also work fine. You can try 2 things at this point:
> 1. Create a new collection with 1 shard and 1 replica and then try adding a
> replica, here's how:
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
>
> > curl --user solr:SolrRocks
>
> http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
>
> This should work fine.
>
> 2. After this, try restarting the solr cluster. Here's how you can do so,
> assuming you didn't change any of the defaults and you are running zk on
> localhost:2181. If not, just change those values below:
> > bin/solr stop -all
>
> After this, check that Solr was actually stopped. I'd also suggest you tail
> the logs on both nodes when they are coming up to see any errors, if any.
> The logs would be here: example/cloud/node1/logs/solr.log
> and example/cloud/node2/logs/solr.log
>
> > bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> > bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
>
> If you get to this checkpoint fine, try adding a read permission.
> Add a permission:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-permission" : {"name":"read",
> "role":"read"}}'
>
> Add a user:
> > curl --user solr:SolrRocks
> http://localhost:8983/solr/admin/authentication
> -H 'Content-type:application/json' -d '{"set-user" :
> {"solrread":"solrRocks"}}'
>
> Assign a role to the user:
> >curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
> -H 'Content-type:application/json' -d '{"set-user-role" :
> {"solrread":["read"]}}'
>
> After this, you should start having issues with ADDREPLICA.
> Also, as you would at this point have a collection with a shard that has a
> replication factor > 1 (remember the ADDREPLICA we did earlier), you would
> have issues when you restart the cluster again using the steps I mentioned
> above.
>
>
> Can you confirm this? I guess I'll just use this text to create a new JIRA
> now.
>
>
> On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > Thank you again for the reply.
> >
> > Below is the Email I was about to send prior to your reply a moment ago:
> > shall I try again without "read" in the security.json?
> >
> >
> >
> > The Collections API method was not discussed in the "Unleashed" class at
> > the conference in DC in 2014 (probably because it was not yet available),
> > so I was using the method I knew.
> >
> > I have now tried again using admin/collections?action=CREATE (using
> > different port numbers to avoid confusion from the failed previous
> > attempts: the previously created nodes had been shutdown and their
> > core.properties files renamed so as not to be discovered), but the
> results
> > are the same:
> > INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting
> > Replication Recovery.
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> > buffering updates.
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> > buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> > INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy;
> Attempting
> > to replicate from http://
> > {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> > ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error
> while
> > trying to
> >
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at http://
> {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> > Expected mime type application/octet-stream but got text/html. <html>
> > <head>
> > <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> > <title>Error 401 Unauthorized request, Response code: 401</title>
> > </head>
> > <body><h2>HTTP ERROR 401</h2>
> > <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> > <pre> Unauthorized request, Response code:
> > 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
> >
> > </body>
> > </html>
> >
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> > at
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> > at
> > org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> > at
> >
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> > at
> > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
> >
> > INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> > buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> > ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery
> > failed - trying again... (2)
> > INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> > x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> > seconds before trying to recover again (3)
> >
> >
> > Below is a list of the steps I took.
> >
> > ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json
> > ~/solr/security151119a.json
> > ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> > ../../solr/configsets/basic_configs/conf -confname xmpl3
> > cd ../../../bin/
> > ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> > ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/
> -z
> > localhost:4545/solr/xmpl3 -s
> > ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> > '
> > curl -u solr:SolrRocks '
> >
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> > '
> >
> >
> >
> >
> > Can you provide a list of steps to take in an out-of-the-box directory
> > tree whereby ADDREPLICA _will_ work with security.json already in place?
> >
> >
> >
> >
> > -----Original Message-----
> > From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> > Sent: Thursday, November 19, 2015 3:44 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > I'll try out what you did later in the day, as soon as I get time but why
> > exactly are you creating cores manually? Seems like you manually create a
> > core and the try to add a replica. Can you try using the Collections API
> to
> > create a collection?
> >
> > Starting Solr 5.0, the only supported way to create a new collection is
> via
> > the Collections API. Creating a core would lead to a collection creation
> > but that's not really supported. It was just something that was done when
> > there were no Collections API.
> >
> >
> > On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> > craig.oakley@nih.gov> wrote:
> >
> > > I tried again with the following security.json, but the results were
> the
> > > same:
> > >
> > > {
> > > "authentication":{
> > > "class":"solr.BasicAuthPlugin",
> > > "credentials":{
> > > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > > "":{"v":9}},
> > > "authorization":{
> > > "class":"solr.RuleBasedAuthorizationPlugin",
> > > "user-role":{
> > > "solr":[
> > > "admin",
> > > "read",
> > > "xmpladmin",
> > > "xmplgen",
> > > "xmplsel"],
> > > "solruser":[
> > > "read",
> > > "xmplgen",
> > > "xmplsel"]},
> > > "permissions":[
> > > {
> > > "name":"security-edit",
> > > "role":"admin"},
> > > {
> > > "name":"xmpl_admin",
> > > "collection":"xmpl",
> > > "path":"/admin/*",
> > > "role":"xmpladmin"},
> > > {
> > > "name":"xmpl_sel",
> > > "collection":"xmpl",
> > > "path":"/select/*",
> > > "role":null},
> > > {
> > > "name":"all-admin",
> > > "collection":null,
> > > "path":"/*",
> > > "role":"xmplgen"},
> > > {
> > > "name":"all-core-handlers",
> > > "path":"/*",
> > > "role":"xmplgen"}],
> > > "":{"v":42}}}
> > >
> > > -----Original Message-----
> > > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > > Sent: Thursday, November 19, 2015 1:46 PM
> > > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> > > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> > >
> > > I note that the thread called "Security Problems" (most recent post by
> > > Nobel Paul) seems like it may help with much of what I'm trying to do.
> I
> > > will see to what extent that may help.
> > >
> >
> >
> >
> > --
> > Anshum Gupta
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thank you for opening SOLR-8326
As a side note, in the procedure you listed, even before adding the collection-admin-edit authorization, I'm already hitting trouble: stopping and restarting a node results in the following
INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4 x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Publishing state of core solr8326_shard2_replica1 as recovering, leader is http://{IP-address-redacted}:8983/solr/solr8326_shard2_replica2/ and I am http://{IP-address-redacted}:7574/solr/solr8326_shard2_replica1/
INFO - 2015-11-20 22:48:41.275; [c:solr8326 s:shard2 r:core_node4 x:solr8326_shard2_replica1] org.apache.solr.cloud.ZkController; publishing state=recovering
INFO - 2015-11-20 22:48:41.278; [c:solr8326 s:shard1 r:core_node3 x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Publishing state of core solr8326_shard1_replica1 as recovering, leader is http://{IP-address-redacted}:8983/solr/solr8326_shard1_replica2/ and I am http://{IP-address-redacted}:7574/solr/solr8326_shard1_replica1/
INFO - 2015-11-20 22:48:41.280; [c:solr8326 s:shard1 r:core_node3 x:solr8326_shard1_replica1] org.apache.solr.cloud.ZkController; publishing state=recovering
INFO - 2015-11-20 22:48:41.282; [c:solr8326 s:shard2 r:core_node4 x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending prep recovery command to http://{IP-address-redacted}:8983/solr; WaitForState: action=PREPRECOVERY&core=solr8326_shard2_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
INFO - 2015-11-20 22:48:41.289; [ ] org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/solr8326/state.json for collection solr8326 has occurred - updating... (live nodes size: 2)
INFO - 2015-11-20 22:48:41.290; [c:solr8326 s:shard1 r:core_node3 x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending prep recovery command to http://{IP-address-redacted}:8983/solr; WaitForState: action=PREPRECOVERY&core=solr8326_shard1_replica2&nodeName={IP-address-redacted}%3A7574_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
INFO - 2015-11-20 22:48:41.291; [ ] org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to ver 25
ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4 x:solr8326_shard2_replica1] org.apache.solr.common.SolrException; Error while trying to recover.:java.util.concurrent.ExecutionException: org.apache.http.ParseException: Invalid content type:
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Caused by: org.apache.http.ParseException: Invalid content type:
at org.apache.http.entity.ContentType.parse(ContentType.java:273)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO - 2015-11-20 22:48:41.298; [ ] org.apache.solr.common.cloud.ZkStateReader$8; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/solr8326/state.json for collection solr8326 has occurred - updating... (live nodes size: 2)
ERROR - 2015-11-20 22:48:41.298; [c:solr8326 s:shard2 r:core_node4 x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (4)
INFO - 2015-11-20 22:48:41.300; [c:solr8326 s:shard2 r:core_node4 x:solr8326_shard2_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait 32.0 seconds before trying to recover again (5)
ERROR - 2015-11-20 22:48:41.300; [c:solr8326 s:shard1 r:core_node3 x:solr8326_shard1_replica1] org.apache.solr.common.SolrException; Error while trying to recover.:java.util.concurrent.ExecutionException: org.apache.http.ParseException: Invalid content type:
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:598)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:361)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Caused by: org.apache.http.ParseException: Invalid content type:
at org.apache.http.entity.ContentType.parse(ContentType.java:273)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:512)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:270)
at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR - 2015-11-20 22:48:41.318; [c:solr8326 s:shard1 r:core_node3 x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (4)
INFO - 2015-11-20 22:48:41.318; [ ] org.apache.solr.common.cloud.ZkStateReader; Updating data for solr8326 to ver 26
INFO - 2015-11-20 22:48:41.319; [c:solr8326 s:shard1 r:core_node3 x:solr8326_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait 32.0 seconds before trying to recover again (5)
I would not be surprised if this were to be some unrelated issue (the symptoms are quite different)
Thanks again
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Friday, November 20, 2015 1:31 PM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Collections API were available before November of 2014, if that is when you
took the class. However, it was only with Solr 5.0 (released in Feb 2015)
that the only supported mechanism to create a collection was restricted to
Collections API.
Here are the list of steps that you'd need to run to see that things are
fine for you without the read permission:
* Untar and setup Solr, don't start it yet
* Start clean zookeeper
* Put the security.json in zk, without anything other than a security-edit
permission. Find the content of the file below. Upload it using your own zk
client or through the solr script:
> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
-cmd putfile /security.json ~/security.json
security.json:
{"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
* Start solr:
> solr-5.3.1/bin/solr start -e cloud -z localhost:2181
You would need to key in a few things e.g. #nodes and ports, leave them at
the default values of 2 nodes and 8983/7574, unless you want to run Solr on
a different port. Then let it create a default collection to just make sure
that everything works fine.
* Add the collection-admin-edit command:
> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
-H 'Content-type:application/json' -d '{"set-permission" :
{"name":"collection-admin-edit", "role":"admin"}}'
At this point, everything should be working fine. Restarting the nodes
should also work fine. You can try 2 things at this point:
1. Create a new collection with 1 shard and 1 replica and then try adding a
replica, here's how:
> curl --user solr:SolrRocks
http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
> curl --user solr:SolrRocks
http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
This should work fine.
2. After this, try restarting the solr cluster. Here's how you can do so,
assuming you didn't change any of the defaults and you are running zk on
localhost:2181. If not, just change those values below:
> bin/solr stop -all
After this, check that Solr was actually stopped. I'd also suggest you tail
the logs on both nodes when they are coming up to see any errors, if any.
The logs would be here: example/cloud/node1/logs/solr.log
and example/cloud/node2/logs/solr.log
> bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
If you get to this checkpoint fine, try adding a read permission.
Add a permission:
> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
-H 'Content-type:application/json' -d '{"set-permission" : {"name":"read",
"role":"read"}}'
Add a user:
> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{"set-user" :
{"solrread":"solrRocks"}}'
Assign a role to the user:
>curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
-H 'Content-type:application/json' -d '{"set-user-role" :
{"solrread":["read"]}}'
After this, you should start having issues with ADDREPLICA.
Also, as you would at this point have a collection with a shard that has a
replication factor > 1 (remember the ADDREPLICA we did earlier), you would
have issues when you restart the cluster again using the steps I mentioned
above.
Can you confirm this? I guess I'll just use this text to create a new JIRA
now.
On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you again for the reply.
>
> Below is the Email I was about to send prior to your reply a moment ago:
> shall I try again without "read" in the security.json?
>
>
>
> The Collections API method was not discussed in the "Unleashed" class at
> the conference in DC in 2014 (probably because it was not yet available),
> so I was using the method I knew.
>
> I have now tried again using admin/collections?action=CREATE (using
> different port numbers to avoid confusion from the failed previous
> attempts: the previously created nodes had been shutdown and their
> core.properties files renamed so as not to be discovered), but the results
> are the same:
> INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting
> Replication Recovery.
> INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> buffering updates.
> INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Attempting
> to replicate from http://
> {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error while
> trying to
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://{IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> Expected mime type application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> <title>Error 401 Unauthorized request, Response code: 401</title>
> </head>
> <body><h2>HTTP ERROR 401</h2>
> <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> <pre> Unauthorized request, Response code:
> 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> at
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>
> INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery
> failed - trying again... (2)
> INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> seconds before trying to recover again (3)
>
>
> Below is a list of the steps I took.
>
> ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json
> ~/solr/security151119a.json
> ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> ../../solr/configsets/basic_configs/conf -confname xmpl3
> cd ../../../bin/
> ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z
> localhost:4545/solr/xmpl3 -s
> ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z
> localhost:4545/solr/xmpl3 -s
> ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> curl -u solr:SolrRocks '
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> '
> curl -u solr:SolrRocks '
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> '
>
>
>
>
> Can you provide a list of steps to take in an out-of-the-box directory
> tree whereby ADDREPLICA _will_ work with security.json already in place?
>
>
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Thursday, November 19, 2015 3:44 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> I'll try out what you did later in the day, as soon as I get time but why
> exactly are you creating cores manually? Seems like you manually create a
> core and the try to add a replica. Can you try using the Collections API to
> create a collection?
>
> Starting Solr 5.0, the only supported way to create a new collection is via
> the Collections API. Creating a core would lead to a collection creation
> but that's not really supported. It was just something that was done when
> there were no Collections API.
>
>
> On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > I tried again with the following security.json, but the results were the
> > same:
> >
> > {
> > "authentication":{
> > "class":"solr.BasicAuthPlugin",
> > "credentials":{
> > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > "":{"v":9}},
> > "authorization":{
> > "class":"solr.RuleBasedAuthorizationPlugin",
> > "user-role":{
> > "solr":[
> > "admin",
> > "read",
> > "xmpladmin",
> > "xmplgen",
> > "xmplsel"],
> > "solruser":[
> > "read",
> > "xmplgen",
> > "xmplsel"]},
> > "permissions":[
> > {
> > "name":"security-edit",
> > "role":"admin"},
> > {
> > "name":"xmpl_admin",
> > "collection":"xmpl",
> > "path":"/admin/*",
> > "role":"xmpladmin"},
> > {
> > "name":"xmpl_sel",
> > "collection":"xmpl",
> > "path":"/select/*",
> > "role":null},
> > {
> > "name":"all-admin",
> > "collection":null,
> > "path":"/*",
> > "role":"xmplgen"},
> > {
> > "name":"all-core-handlers",
> > "path":"/*",
> > "role":"xmplgen"}],
> > "":{"v":42}}}
> >
> > -----Original Message-----
> > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > Sent: Thursday, November 19, 2015 1:46 PM
> > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > I note that the thread called "Security Problems" (most recent post by
> > Nobel Paul) seems like it may help with much of what I'm trying to do. I
> > will see to what extent that may help.
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
Collections API were available before November of 2014, if that is when you
took the class. However, it was only with Solr 5.0 (released in Feb 2015)
that the only supported mechanism to create a collection was restricted to
Collections API.
Here are the list of steps that you'd need to run to see that things are
fine for you without the read permission:
* Untar and setup Solr, don't start it yet
* Start clean zookeeper
* Put the security.json in zk, without anything other than a security-edit
permission. Find the content of the file below. Upload it using your own zk
client or through the solr script:
> solr-5.3.1/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181
-cmd putfile /security.json ~/security.json
security.json:
{"authentication":{"class":"solr.BasicAuthPlugin","credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},"authorization":{"class":"solr.RuleBasedAuthorizationPlugin","user-role":{"solr":["admin"]},"permissions":[{"name":"security-edit","role":"admin"}]}}
* Start solr:
> solr-5.3.1/bin/solr start -e cloud -z localhost:2181
You would need to key in a few things e.g. #nodes and ports, leave them at
the default values of 2 nodes and 8983/7574, unless you want to run Solr on
a different port. Then let it create a default collection to just make sure
that everything works fine.
* Add the collection-admin-edit command:
> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
-H 'Content-type:application/json' -d '{"set-permission" :
{"name":"collection-admin-edit", "role":"admin"}}'
At this point, everything should be working fine. Restarting the nodes
should also work fine. You can try 2 things at this point:
1. Create a new collection with 1 shard and 1 replica and then try adding a
replica, here's how:
> curl --user solr:SolrRocks
http://localhost:8983/solr/admin/collections?action=CREATE&name=testcollection&collection.configName=gettingstarted&numShards=1
> curl --user solr:SolrRocks
http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=testcollection&shard=shard1
This should work fine.
2. After this, try restarting the solr cluster. Here's how you can do so,
assuming you didn't change any of the defaults and you are running zk on
localhost:2181. If not, just change those values below:
> bin/solr stop -all
After this, check that Solr was actually stopped. I'd also suggest you tail
the logs on both nodes when they are coming up to see any errors, if any.
The logs would be here: example/cloud/node1/logs/solr.log
and example/cloud/node2/logs/solr.log
> bin/solr start -c -p 8983 -s "example/cloud/node1/solr" -z localhost:2181
> bin/solr start -c -p 7574 -s "example/cloud/node2/solr" -z localhost:2181
If you get to this checkpoint fine, try adding a read permission.
Add a permission:
> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
-H 'Content-type:application/json' -d '{"set-permission" : {"name":"read",
"role":"read"}}'
Add a user:
> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{"set-user" :
{"solrread":"solrRocks"}}'
Assign a role to the user:
>curl --user solr:SolrRocks http://localhost:8983/solr/admin/authorization
-H 'Content-type:application/json' -d '{"set-user-role" :
{"solrread":["read"]}}'
After this, you should start having issues with ADDREPLICA.
Also, as you would at this point have a collection with a shard that has a
replication factor > 1 (remember the ADDREPLICA we did earlier), you would
have issues when you restart the cluster again using the steps I mentioned
above.
Can you confirm this? I guess I'll just use this text to create a new JIRA
now.
On Fri, Nov 20, 2015 at 10:04 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Thank you again for the reply.
>
> Below is the Email I was about to send prior to your reply a moment ago:
> shall I try again without "read" in the security.json?
>
>
>
> The Collections API method was not discussed in the "Unleashed" class at
> the conference in DC in 2014 (probably because it was not yet available),
> so I was using the method I knew.
>
> I have now tried again using admin/collections?action=CREATE (using
> different port numbers to avoid confusion from the failed previous
> attempts: the previously created nodes had been shutdown and their
> core.properties files renamed so as not to be discovered), but the results
> are the same:
> INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting
> Replication Recovery.
> INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin
> buffering updates.
> INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to
> buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Attempting
> to replicate from http://
> {IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
> ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error while
> trying to
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://{IP-address-redacted}:4685/solr/xmpl3_shard1_replica1:
> Expected mime type application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> <title>Error 401 Unauthorized request, Response code: 401</title>
> </head>
> <body><h2>HTTP ERROR 401</h2>
> <p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
> <pre> Unauthorized request, Response code:
> 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> at
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>
> INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping
> buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
> ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery
> failed - trying again... (2)
> INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2
> x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> seconds before trying to recover again (3)
>
>
> Below is a list of the steps I took.
>
> ./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
> ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json
> ~/solr/security151119a.json
> ./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir
> ../../solr/configsets/basic_configs/conf -confname xmpl3
> cd ../../../bin/
> ./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z
> localhost:4545/solr/xmpl3 -s
> ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
> ./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z
> localhost:4545/solr/xmpl3 -s
> ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
> curl -u solr:SolrRocks '
> http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr
> '
> curl -u solr:SolrRocks '
> http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true
> '
>
>
>
>
> Can you provide a list of steps to take in an out-of-the-box directory
> tree whereby ADDREPLICA _will_ work with security.json already in place?
>
>
>
>
> -----Original Message-----
> From: Anshum Gupta [mailto:anshum@anshumgupta.net]
> Sent: Thursday, November 19, 2015 3:44 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> I'll try out what you did later in the day, as soon as I get time but why
> exactly are you creating cores manually? Seems like you manually create a
> core and the try to add a replica. Can you try using the Collections API to
> create a collection?
>
> Starting Solr 5.0, the only supported way to create a new collection is via
> the Collections API. Creating a core would lead to a collection creation
> but that's not really supported. It was just something that was done when
> there were no Collections API.
>
>
> On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oakley@nih.gov> wrote:
>
> > I tried again with the following security.json, but the results were the
> > same:
> >
> > {
> > "authentication":{
> > "class":"solr.BasicAuthPlugin",
> > "credentials":{
> > "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> > "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> > 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> > "":{"v":9}},
> > "authorization":{
> > "class":"solr.RuleBasedAuthorizationPlugin",
> > "user-role":{
> > "solr":[
> > "admin",
> > "read",
> > "xmpladmin",
> > "xmplgen",
> > "xmplsel"],
> > "solruser":[
> > "read",
> > "xmplgen",
> > "xmplsel"]},
> > "permissions":[
> > {
> > "name":"security-edit",
> > "role":"admin"},
> > {
> > "name":"xmpl_admin",
> > "collection":"xmpl",
> > "path":"/admin/*",
> > "role":"xmpladmin"},
> > {
> > "name":"xmpl_sel",
> > "collection":"xmpl",
> > "path":"/select/*",
> > "role":null},
> > {
> > "name":"all-admin",
> > "collection":null,
> > "path":"/*",
> > "role":"xmplgen"},
> > {
> > "name":"all-core-handlers",
> > "path":"/*",
> > "role":"xmplgen"}],
> > "":{"v":42}}}
> >
> > -----Original Message-----
> > From: Oakley, Craig (NIH/NLM/NCBI) [C]
> > Sent: Thursday, November 19, 2015 1:46 PM
> > To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> > Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
> >
> > I note that the thread called "Security Problems" (most recent post by
> > Nobel Paul) seems like it may help with much of what I'm trying to do. I
> > will see to what extent that may help.
> >
>
>
>
> --
> Anshum Gupta
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thank you again for the reply.
Below is the Email I was about to send prior to your reply a moment ago: shall I try again without "read" in the security.json?
The Collections API method was not discussed in the "Unleashed" class at the conference in DC in 2014 (probably because it was not yet available), so I was using the method I knew.
I have now tried again using admin/collections?action=CREATE (using different port numbers to avoid confusion from the failed previous attempts: the previously created nodes had been shutdown and their core.properties files renamed so as not to be discovered), but the results are the same:
INFO - 2015-11-20 16:56:25.283; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery.
INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates.
INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
INFO - 2015-11-20 16:56:25.284; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://{IP-address-redacted}:4685/solr/xmpl3_shard1_replica1/.
ERROR - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.common.SolrException; Error while trying to recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://{IP-address-redacted}:4685/solr/xmpl3_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Unauthorized request, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/xmpl3_shard1_replica1/update. Reason:
<pre> Unauthorized request, Response code: 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
INFO - 2015-11-20 16:56:25.292; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.update.UpdateLog; Dropping buffered updates FSUpdateLog{state=BUFFERING, tlog=null}
ERROR - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (2)
INFO - 2015-11-20 16:56:25.293; [c:xmpl3 s:shard1 r:core_node2 x:xmpl3_shard1_replica2] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0 seconds before trying to recover again (3)
Below is a list of the steps I took.
./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl3
./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd putfile /security.json ~/solr/security151119a.json
./zkcli.sh --zkhost localhost:4545/solr/xmpl3 -cmd upconfig -confdir ../../solr/configsets/basic_configs/conf -confname xmpl3
cd ../../../bin/
./solr -c -p 4695 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z localhost:4545/solr/xmpl3 -s ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
./solr -c -p 4685 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z localhost:4545/solr/xmpl3 -s ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
curl -u solr:SolrRocks 'http://nosqltest11:4685/solr/admin/collections?action=CREATE&name=xmpl3&numShards=1&replicationFactor=1&createNodeSet={IP-address-redacted}:4685_solr'
curl -u solr:SolrRocks 'http://nosqltest11:4685/solr/admin/collections?action=ADDREPLICA&collection=xmpl3&shard=shard1&node={IP-address-redacted}:4695_solr&wt=json&indent=true'
Can you provide a list of steps to take in an out-of-the-box directory tree whereby ADDREPLICA _will_ work with security.json already in place?
-----Original Message-----
From: Anshum Gupta [mailto:anshum@anshumgupta.net]
Sent: Thursday, November 19, 2015 3:44 PM
To: solr-user@lucene.apache.org
Subject: Re: Re:Re: Implementing security.json is breaking ADDREPLICA
I'll try out what you did later in the day, as soon as I get time but why
exactly are you creating cores manually? Seems like you manually create a
core and the try to add a replica. Can you try using the Collections API to
create a collection?
Starting Solr 5.0, the only supported way to create a new collection is via
the Collections API. Creating a core would lead to a collection creation
but that's not really supported. It was just something that was done when
there were no Collections API.
On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> I tried again with the following security.json, but the results were the
> same:
>
> {
> "authentication":{
> "class":"solr.BasicAuthPlugin",
> "credentials":{
> "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> "":{"v":9}},
> "authorization":{
> "class":"solr.RuleBasedAuthorizationPlugin",
> "user-role":{
> "solr":[
> "admin",
> "read",
> "xmpladmin",
> "xmplgen",
> "xmplsel"],
> "solruser":[
> "read",
> "xmplgen",
> "xmplsel"]},
> "permissions":[
> {
> "name":"security-edit",
> "role":"admin"},
> {
> "name":"xmpl_admin",
> "collection":"xmpl",
> "path":"/admin/*",
> "role":"xmpladmin"},
> {
> "name":"xmpl_sel",
> "collection":"xmpl",
> "path":"/select/*",
> "role":null},
> {
> "name":"all-admin",
> "collection":null,
> "path":"/*",
> "role":"xmplgen"},
> {
> "name":"all-core-handlers",
> "path":"/*",
> "role":"xmplgen"}],
> "":{"v":42}}}
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Thursday, November 19, 2015 1:46 PM
> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> I note that the thread called "Security Problems" (most recent post by
> Nobel Paul) seems like it may help with much of what I'm trying to do. I
> will see to what extent that may help.
>
--
Anshum Gupta
Re: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
I'll try out what you did later in the day, as soon as I get time but why
exactly are you creating cores manually? Seems like you manually create a
core and the try to add a replica. Can you try using the Collections API to
create a collection?
Starting Solr 5.0, the only supported way to create a new collection is via
the Collections API. Creating a core would lead to a collection creation
but that's not really supported. It was just something that was done when
there were no Collections API.
On Thu, Nov 19, 2015 at 12:36 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> I tried again with the following security.json, but the results were the
> same:
>
> {
> "authentication":{
> "class":"solr.BasicAuthPlugin",
> "credentials":{
> "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> "":{"v":9}},
> "authorization":{
> "class":"solr.RuleBasedAuthorizationPlugin",
> "user-role":{
> "solr":[
> "admin",
> "read",
> "xmpladmin",
> "xmplgen",
> "xmplsel"],
> "solruser":[
> "read",
> "xmplgen",
> "xmplsel"]},
> "permissions":[
> {
> "name":"security-edit",
> "role":"admin"},
> {
> "name":"xmpl_admin",
> "collection":"xmpl",
> "path":"/admin/*",
> "role":"xmpladmin"},
> {
> "name":"xmpl_sel",
> "collection":"xmpl",
> "path":"/select/*",
> "role":null},
> {
> "name":"all-admin",
> "collection":null,
> "path":"/*",
> "role":"xmplgen"},
> {
> "name":"all-core-handlers",
> "path":"/*",
> "role":"xmplgen"}],
> "":{"v":42}}}
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C]
> Sent: Thursday, November 19, 2015 1:46 PM
> To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
> Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
>
> I note that the thread called "Security Problems" (most recent post by
> Nobel Paul) seems like it may help with much of what I'm trying to do. I
> will see to what extent that may help.
>
--
Anshum Gupta
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
I tried again with the following security.json, but the results were the same:
{
"authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":{
"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
"solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE= 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
"":{"v":9}},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{
"solr":[
"admin",
"read",
"xmpladmin",
"xmplgen",
"xmplsel"],
"solruser":[
"read",
"xmplgen",
"xmplsel"]},
"permissions":[
{
"name":"security-edit",
"role":"admin"},
{
"name":"xmpl_admin",
"collection":"xmpl",
"path":"/admin/*",
"role":"xmpladmin"},
{
"name":"xmpl_sel",
"collection":"xmpl",
"path":"/select/*",
"role":null},
{
"name":"all-admin",
"collection":null,
"path":"/*",
"role":"xmplgen"},
{
"name":"all-core-handlers",
"path":"/*",
"role":"xmplgen"}],
"":{"v":42}}}
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C]
Sent: Thursday, November 19, 2015 1:46 PM
To: 'solr-user@lucene.apache.org' <so...@lucene.apache.org>
Subject: RE: Re:Re: Implementing security.json is breaking ADDREPLICA
I note that the thread called "Security Problems" (most recent post by Nobel Paul) seems like it may help with much of what I'm trying to do. I will see to what extent that may help.
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
I note that the thread called "Security Problems" (most recent post by Nobel Paul) seems like it may help with much of what I'm trying to do. I will see to what extent that may help.
RE: Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov>.
Thank you for the reply.
What we are attempting is to require a password for practically everything, so that even were a hacker to get within the firewall, they would have limited access to the various services (the Security team even complained that, for Solr 4.5 servers, attempts to access host:port (without "/solr") resulted in an error message that gave the full pathname to solr.war)
I am sending the solr.log files directly to Anshum, so as not to clutter up the Email list.
The steps I used to recreate the problem are as follows:
cd zookeeper-3.4.6/conf/
sed 's/2181/4545/' zoo_sample.cfg | tee zoo_sample4545.cfg
cd ../bin
./zkServer.sh start zoo_sample4545.cfg
cd ../../solr-5.3.1/server/solr
mkdir xmpl
echo 'name=xmpl' | tee xmpl/core.properties
mkdir xmpl/data
mkdir xmpl/data/index
mkdir xmpl/data/tlog
cd ../scripts/cloud-scripts/
./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr
./zkcli.sh --zkhost localhost:4545 -cmd makepath /solr/xmpl
./zkcli.sh --zkhost localhost:4545/solr/xmpl -cmd upconfig -confdir ../../solr/configsets/basic_configs/conf -confname xmpl
mkdir ../../example/solr
cp solr.xml ../../example/solr
./zkcli.sh --zkhost localhost:4545/solr/xmpl -cmd putfile /security.json ~/solr/security151117a.json
cd ../../../bin
mkdir ../example/solr/pid
./solr -c -p 4575 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z localhost:4545/solr/xmpl -s ~dbman/solr/straight531outofbox/solr-5.3.1/example/solr
./solr -c -p 4565 -d ~dbman/solr/straight531outofbox/solr-5.3.1/server/ -z localhost:4545/solr/xmpl -s ~dbman/solr/straight531outofbox/solr-5.3.1/server/solr
curl -u solr:SolrRocks 'http:// {IP-address-redacted}:4575/solr/admin/collections?action=ADDREPLICA&collection=xmpl&shard=shard1&node={IP-address-redacted}:4575_solr&wt=json&indent=true'
The contents of security151117a.json is included in the original post
If there is a better way using the Well Known Permissions as described at lucidworks.com/blog/2015/08/17/securing-solr-basic-auth-permission-rules, I'm open to trying that.
I would like to acknowledge that there definitely seem to be some IMPROVEMENTS in the security.json implementation: particularly in terms of Core Admin (using jetty-implemented Authentication in webdefault.xml, anyone who could get into the GUI front page could rename cores, unless prevented by OS-level permissions on core.properties).
Thanks again
Re:Re: Implementing security.json is breaking ADDREPLICA
Posted by 马柏樟 <ma...@126.com>.
Hi Anshum,
I encounter the same problem after I config my security.json like this:
{ "authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{"solr":"admin"},
"permissions":[
{ "name":"security-edit",
"role":"admin"},
{ "name":"read",
"role":"admin"},
{ "name":"update",
"role":"admin"}],
"":{"v":3}}}
I just want to restrict strangers to do select/create/update operations on my collections and configs like schema.xml or other things related to solr itself from both Admin UI and sorj. But it is useless and error occurs like this:
ERROR (RecoveryThread-gettingstarted_shard2_replica2) [c:gettingstarted s:shard2 r:core_node2 x:gettingstarted_shard2_replica2] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://(myhost):7574/solr/gettingstarted_shard2_replica1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Unauthorized request, Response code: 401</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/gettingstarted_shard2_replica1/update. Reason:
<pre> Unauthorized request, Response code: 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Error while trying to recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://(myhost):7574/solr/gettingstarted_shard2_replica1: Expected mime type application/octet-stream but got text/html. <html>
Kind regards
Byzen Ma
At 2015-11-19 13:46:08, "Anshum Gupta" <an...@anshumgupta.net> wrote:
>Hi Craig,
>
>Just to be sure that you're using the feature as it should be used, can you
>outline what is it that you're trying to do here? There are a few things
>that aren't clear to me here, e.g. I see permissions for the /admin handler
>for a particular collection.
>
>What are the kind of permissions you're trying to set up.
>
>Solr uses it's internal PKI based mechanism for inter-shard communication
>and so you shouldn't really be hitting this. Can you check your logs and
>tell me if there are any other exceptions you see while bringing the node
>up etc. ? Something from PKI itself.
>
>About restricting the UI, there's another thread in parallel that's been
>discussing exactly that. The thing with the current UI implementation is
>that it bypasses all of this, primarily because most of that content is
>static. I am not saying we should be able to put it behind the
>authentication layer, but just that it's not currently supported through
>this plugin.
>
>On Wed, Nov 18, 2015 at 11:20 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>craig.oakley@nih.gov> wrote:
>
>> Implementing security.json is breaking ADDREPLICA
>>
>> I have been able to reproduce this issue with minimal changes from an
>> out-of-the-box Zookeeper (3.4.6) and Solr (5.3.1): loading
>> configsets/basic_configs/conf into Zookeeper, creating the security.json
>> listed below, creating two nodes (one with a core named xmpl and one
>> without any core)- I can provide details if helpful.
>>
>> The security.json is as follows:
>>
>> {
>> "authentication":{
>> "class":"solr.BasicAuthPlugin",
>> "credentials":{
>> "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
>> "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
>> 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
>> "":{"v":9}},
>> "authorization":{
>> "class":"solr.RuleBasedAuthorizationPlugin",
>> "user-role":{
>> "solr":[
>> "admin",
>> "read",
>> "xmpladmin",
>> "xmplgen",
>> "xmplsel"],
>> "solruser":[
>> "read",
>> "xmplgen",
>> "xmplsel"]},
>> "permissions":[
>> {
>> "name":"security-edit",
>> "role":"admin"},
>> {
>> "name":"xmpl_admin",
>> "collection":"xmpl",
>> "path":"/admin/*",
>> "role":"xmpladmin"},
>> {
>> "name":"xmpl_sel",
>> "collection":"xmpl",
>> "path":"/select/*",
>> "role":null},
>> {
>> "name":"xmpl_gen",
>> "collection":"xmpl",
>> "path":"/*",
>> "role":"xmplgen"}],
>> "":{"v":42}}}
>>
>>
>>
>>
>>
>> When I then execute admin/collections?action=ADDREPLICA, I get errors such
>> as the following in the solr.log of the node which was created without a
>> core.
>>
>> INFO - 2015-11-17 21:03:54.157; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Starting
>> Replication Recovery.
>> INFO - 2015-11-17 21:03:54.158; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Begin
>> buffering updates.
>> INFO - 2015-11-17 21:03:54.158; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.update.UpdateLog; Starting to
>> buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
>> INFO - 2015-11-17 21:03:54.159; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Attempting
>> to replicate from http://{IP-address-redacted}:4565/solr/xmpl/.
>> ERROR - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.common.SolrException; Error while
>> trying to
>> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> Error from server at http://{IP-address-redacted}:4565/solr/xmpl:
>> Expected mime type application/octet-stream but got text/html. <html>
>> <head>
>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>> <title>Error 401 Unauthorized request, Response code: 401</title>
>> </head>
>> <body><h2>HTTP ERROR 401</h2>
>> <p>Problem accessing /solr/xmpl/update. Reason:
>> <pre> Unauthorized request, Response code:
>> 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>>
>> </body>
>> </html>
>>
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
>> at
>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
>> at
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
>> at
>> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
>> at
>> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>>
>> INFO - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.update.UpdateLog; Dropping buffered
>> updates FSUpdateLog{state=BUFFERING, tlog=null}
>> ERROR - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Recovery
>> failed - trying again... (2)
>> INFO - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
>> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
>> seconds before trying to recover again (3)
>>
>>
>>
>> And (after modifying Logging Levels), the solr.log of the node which
>> already had a core gets errors such as the following:
>>
>> 2015-11-17 21:03:50.743 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
>> REQUEST GET /solr/tpl/cloud.html on HttpChannelOverHttp@37cf94f4
>> {r=1,c=false,a=DISPATCHED,uri=/solr/tpl/cloud.html}
>> 2015-11-17 21:03:50.744 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
>> RESPONSE /solr/tpl/cloud.html 200 handled=true
>> 2015-11-17 21:03:50.802 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
>> REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4
>> {r=2,c=false,a=DISPATCHED,uri=/solr/zookeeper}
>> 2015-11-17 21:03:50.803 INFO (qtp59559151-91) [ ] o.a.s.s.HttpSolrCall
>> userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
>> 2015-11-17 21:03:50.831 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
>> RESPONSE /solr/zookeeper 200 handled=true
>> 2015-11-17 21:03:50.837 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
>> REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4
>> {r=3,c=false,a=DISPATCHED,uri=/solr/zookeeper}
>> 2015-11-17 21:03:50.838 INFO (qtp59559151-87) [ ] o.a.s.s.HttpSolrCall
>> userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
>> 2015-11-17 21:03:50.841 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
>> RESPONSE /solr/zookeeper 200 handled=true
>> 2015-11-17 21:03:50.857 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
>> REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4
>> {r=4,c=false,a=DISPATCHED,uri=/solr/zookeeper}
>> 2015-11-17 21:03:50.858 INFO (qtp59559151-91) [ ] o.a.s.s.HttpSolrCall
>> userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
>> 2015-11-17 21:03:50.860 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
>> RESPONSE /solr/zookeeper 200 handled=true
>> 2015-11-17 21:03:54.162 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
>> REQUEST POST /solr/xmpl/update on HttpChannelOverHttp@1cf967f0
>> {r=1,c=false,a=DISPATCHED,uri=/solr/xmpl/update}
>> 2015-11-17 21:03:54.164 INFO (qtp59559151-87) [c:xmpl s:shard1
>> r:core_node1 x:xmpl] o.a.s.s.HttpSolrCall userPrincipal: [null] type:
>> [WRITE], collections: [xmpl,], Path: [/update]
>> 2015-11-17 21:03:54.164 DEBUG (qtp59559151-87) [c:xmpl s:shard1
>> r:core_node1 x:xmpl] o.e.j.s.Server RESPONSE /solr/xmpl/update 401
>> handled=true
>>
>>
>>
>> My impression from Anshum Gupta's 10/16/15 talk in Austin at the Solr
>> conference was that this was supposed to work. It does seem that one might
>> be able to add security to replication, but there does not seem to be a way
>> to add SolrCloud replication to this type of security.
>>
>> Also, on a side note, I notice that http://hostname:port/solr/ does bring
>> up the GUI without prompting for a password: the Security team here would
>> like us to implement security.json in such a way that even the front page
>> of the GUI will require a password (although they will allow us to allow
>> select access without a password): I have not yet found a way via
>> security.json to implement that a password would be required in order to
>> access the GUI front page.
>>
>>
>>
>> Please advise.
>>
>>
>
>
>--
>Anshum Gupta
Re: Implementing security.json is breaking ADDREPLICA
Posted by Anshum Gupta <an...@anshumgupta.net>.
Hi Craig,
Just to be sure that you're using the feature as it should be used, can you
outline what is it that you're trying to do here? There are a few things
that aren't clear to me here, e.g. I see permissions for the /admin handler
for a particular collection.
What are the kind of permissions you're trying to set up.
Solr uses it's internal PKI based mechanism for inter-shard communication
and so you shouldn't really be hitting this. Can you check your logs and
tell me if there are any other exceptions you see while bringing the node
up etc. ? Something from PKI itself.
About restricting the UI, there's another thread in parallel that's been
discussing exactly that. The thing with the current UI implementation is
that it bypasses all of this, primarily because most of that content is
static. I am not saying we should be able to put it behind the
authentication layer, but just that it's not currently supported through
this plugin.
On Wed, Nov 18, 2015 at 11:20 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oakley@nih.gov> wrote:
> Implementing security.json is breaking ADDREPLICA
>
> I have been able to reproduce this issue with minimal changes from an
> out-of-the-box Zookeeper (3.4.6) and Solr (5.3.1): loading
> configsets/basic_configs/conf into Zookeeper, creating the security.json
> listed below, creating two nodes (one with a core named xmpl and one
> without any core)- I can provide details if helpful.
>
> The security.json is as follows:
>
> {
> "authentication":{
> "class":"solr.BasicAuthPlugin",
> "credentials":{
> "solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=",
> "solruser":"VgZX1TAMNHT2IJikoGdKtxQdXc+MbNwfqzf89YqcLEE=
> 37pPWQ9v4gciIKHuTmFmN0Rv66rnlMOFEWfEy9qjJfY="},
> "":{"v":9}},
> "authorization":{
> "class":"solr.RuleBasedAuthorizationPlugin",
> "user-role":{
> "solr":[
> "admin",
> "read",
> "xmpladmin",
> "xmplgen",
> "xmplsel"],
> "solruser":[
> "read",
> "xmplgen",
> "xmplsel"]},
> "permissions":[
> {
> "name":"security-edit",
> "role":"admin"},
> {
> "name":"xmpl_admin",
> "collection":"xmpl",
> "path":"/admin/*",
> "role":"xmpladmin"},
> {
> "name":"xmpl_sel",
> "collection":"xmpl",
> "path":"/select/*",
> "role":null},
> {
> "name":"xmpl_gen",
> "collection":"xmpl",
> "path":"/*",
> "role":"xmplgen"}],
> "":{"v":42}}}
>
>
>
>
>
> When I then execute admin/collections?action=ADDREPLICA, I get errors such
> as the following in the solr.log of the node which was created without a
> core.
>
> INFO - 2015-11-17 21:03:54.157; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Starting
> Replication Recovery.
> INFO - 2015-11-17 21:03:54.158; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Begin
> buffering updates.
> INFO - 2015-11-17 21:03:54.158; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.update.UpdateLog; Starting to
> buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> INFO - 2015-11-17 21:03:54.159; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Attempting
> to replicate from http://{IP-address-redacted}:4565/solr/xmpl/.
> ERROR - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.common.SolrException; Error while
> trying to
> recover:org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://{IP-address-redacted}:4565/solr/xmpl:
> Expected mime type application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> <title>Error 401 Unauthorized request, Response code: 401</title>
> </head>
> <body><h2>HTTP ERROR 401</h2>
> <p>Problem accessing /solr/xmpl/update. Reason:
> <pre> Unauthorized request, Response code:
> 401</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
>
> </body>
> </html>
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:528)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:234)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:226)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
> at
> org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:207)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:147)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
>
> INFO - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.update.UpdateLog; Dropping buffered
> updates FSUpdateLog{state=BUFFERING, tlog=null}
> ERROR - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Recovery
> failed - trying again... (2)
> INFO - 2015-11-17 21:03:54.166; [c:xmpl s:shard1 r:core_node2
> x:xmpl_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait 8.0
> seconds before trying to recover again (3)
>
>
>
> And (after modifying Logging Levels), the solr.log of the node which
> already had a core gets errors such as the following:
>
> 2015-11-17 21:03:50.743 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
> REQUEST GET /solr/tpl/cloud.html on HttpChannelOverHttp@37cf94f4
> {r=1,c=false,a=DISPATCHED,uri=/solr/tpl/cloud.html}
> 2015-11-17 21:03:50.744 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
> RESPONSE /solr/tpl/cloud.html 200 handled=true
> 2015-11-17 21:03:50.802 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
> REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4
> {r=2,c=false,a=DISPATCHED,uri=/solr/zookeeper}
> 2015-11-17 21:03:50.803 INFO (qtp59559151-91) [ ] o.a.s.s.HttpSolrCall
> userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
> 2015-11-17 21:03:50.831 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
> RESPONSE /solr/zookeeper 200 handled=true
> 2015-11-17 21:03:50.837 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
> REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4
> {r=3,c=false,a=DISPATCHED,uri=/solr/zookeeper}
> 2015-11-17 21:03:50.838 INFO (qtp59559151-87) [ ] o.a.s.s.HttpSolrCall
> userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
> 2015-11-17 21:03:50.841 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
> RESPONSE /solr/zookeeper 200 handled=true
> 2015-11-17 21:03:50.857 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
> REQUEST GET /solr/zookeeper on HttpChannelOverHttp@37cf94f4
> {r=4,c=false,a=DISPATCHED,uri=/solr/zookeeper}
> 2015-11-17 21:03:50.858 INFO (qtp59559151-91) [ ] o.a.s.s.HttpSolrCall
> userPrincipal: [null] type: [UNKNOWN], collections: [], Path: [/zookeeper]
> 2015-11-17 21:03:50.860 DEBUG (qtp59559151-91) [ ] o.e.j.s.Server
> RESPONSE /solr/zookeeper 200 handled=true
> 2015-11-17 21:03:54.162 DEBUG (qtp59559151-87) [ ] o.e.j.s.Server
> REQUEST POST /solr/xmpl/update on HttpChannelOverHttp@1cf967f0
> {r=1,c=false,a=DISPATCHED,uri=/solr/xmpl/update}
> 2015-11-17 21:03:54.164 INFO (qtp59559151-87) [c:xmpl s:shard1
> r:core_node1 x:xmpl] o.a.s.s.HttpSolrCall userPrincipal: [null] type:
> [WRITE], collections: [xmpl,], Path: [/update]
> 2015-11-17 21:03:54.164 DEBUG (qtp59559151-87) [c:xmpl s:shard1
> r:core_node1 x:xmpl] o.e.j.s.Server RESPONSE /solr/xmpl/update 401
> handled=true
>
>
>
> My impression from Anshum Gupta's 10/16/15 talk in Austin at the Solr
> conference was that this was supposed to work. It does seem that one might
> be able to add security to replication, but there does not seem to be a way
> to add SolrCloud replication to this type of security.
>
> Also, on a side note, I notice that http://hostname:port/solr/ does bring
> up the GUI without prompting for a password: the Security team here would
> like us to implement security.json in such a way that even the front page
> of the GUI will require a password (although they will allow us to allow
> select access without a password): I have not yet found a way via
> security.json to implement that a password would be required in order to
> access the GUI front page.
>
>
>
> Please advise.
>
>
--
Anshum Gupta