You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Sundaram, Dinesh" <Di...@mastercard.com> on 2017/12/11 19:24:23 UTC
Solr ssl issue while creating collection
Hi,
How do I change the protocol to https everywhere including replica.
NOTE: I have just only one node 8983. started solr using this command.
bin/solr start -cloud -p 8983 -noprompt
1. Configure SSL using https://lucene.apache.org/solr/guide/7_1/enabling-ssl.html
2. Restart solr
3. Validate solr with https url https://localhost:8983/solr - works fine
4. Create a collection https://localhost:8983/solr/#/~collections
5. here is the response :
Connection to Solr lost
Please check the Solr instance.
6.Server solr.log: here notice the replica call goes to http port instead of https
2017-12-11 11:52:27.929 ERROR (OverseerThreadFactory-8-thread-1-processing-n:localhost:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://localhost:8983/solr
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:8983/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:640)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
... 12 more
Caused by: org.apache.http.ProtocolException: The server failed to respond with a valid HTTP response
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:149)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
... 15 more
Dinesh Sundaram
MBS Platform Engineering
Mastercard
[cid:image001.png@01D37283.5B72AA80]<www.mastercard.com>
CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
RE: Solr ssl issue while creating collection
Posted by "Sundaram, Dinesh" <Di...@mastercard.com>.
Thanks Erick for your valuable reply. Much Appreciated !!!
Dinesh Sundaram
MBS Platform Engineering
Mastercard
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Friday, December 15, 2017 5:17 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Solr ssl issue while creating collection
No. ZooKeeper is an integral part of SolrCloud, without it you don't _have_ SolrCloud.
Best,
Erick
On Fri, Dec 15, 2017 at 1:03 PM, Sundaram, Dinesh <Di...@mastercard.com> wrote:
> Thanks again for your valuable reply. Yes that’s correct. Is there a way to start solr alone without any embedded/external zookeeper in solrcloud mode?
>
>
> Dinesh Sundaram
> MBS Platform Engineering
>
> Mastercard
>
>
>
> -----Original Message-----
> From: Shawn Heisey [mailto:apache@elyograg.org]
> Sent: Wednesday, December 13, 2017 4:54 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr ssl issue while creating collection
>
> On 12/13/2017 3:16 PM, Sundaram, Dinesh wrote:
>> Thanks Shawn for your input, Is this errors specific only for zookeeper operations? If so is there any way to turn off default zookeeper which runs on 9983?
>
> If you don't want to start the embedded zookeeper, then you want to be sure that you have a zkHost defined which lists all of the hosts in your external ensemble. You can either define ZK_HOST in the include script, or use the -z option when starting Solr manually. When Solr is provided with information about ZK hosts, it does NOT start the embedded ZK.
>
> The exceptions you're seeing have nothing to do with zookeeper. The latest exception you mentioned is caused by one SolrCloud instance sending HTTPS requests to another SolrCloud instance, and failing to validate SSL because the hostname doesn't match the info in the certificate.
>
> Thanks,
> Shawn
>
>
> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Re: Solr ssl issue while creating collection
Posted by Erick Erickson <er...@gmail.com>.
No. ZooKeeper is an integral part of SolrCloud, without it you don't
_have_ SolrCloud.
Best,
Erick
On Fri, Dec 15, 2017 at 1:03 PM, Sundaram, Dinesh
<Di...@mastercard.com> wrote:
> Thanks again for your valuable reply. Yes that’s correct. Is there a way to start solr alone without any embedded/external zookeeper in solrcloud mode?
>
>
> Dinesh Sundaram
> MBS Platform Engineering
>
> Mastercard
>
>
>
> -----Original Message-----
> From: Shawn Heisey [mailto:apache@elyograg.org]
> Sent: Wednesday, December 13, 2017 4:54 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr ssl issue while creating collection
>
> On 12/13/2017 3:16 PM, Sundaram, Dinesh wrote:
>> Thanks Shawn for your input, Is this errors specific only for zookeeper operations? If so is there any way to turn off default zookeeper which runs on 9983?
>
> If you don't want to start the embedded zookeeper, then you want to be sure that you have a zkHost defined which lists all of the hosts in your external ensemble. You can either define ZK_HOST in the include script, or use the -z option when starting Solr manually. When Solr is provided with information about ZK hosts, it does NOT start the embedded ZK.
>
> The exceptions you're seeing have nothing to do with zookeeper. The latest exception you mentioned is caused by one SolrCloud instance sending HTTPS requests to another SolrCloud instance, and failing to validate SSL because the hostname doesn't match the info in the certificate.
>
> Thanks,
> Shawn
>
>
> CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
RE: Solr ssl issue while creating collection
Posted by "Sundaram, Dinesh" <Di...@mastercard.com>.
Thanks again for your valuable reply. Yes that’s correct. Is there a way to start solr alone without any embedded/external zookeeper in solrcloud mode?
Dinesh Sundaram
MBS Platform Engineering
Mastercard
-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Wednesday, December 13, 2017 4:54 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr ssl issue while creating collection
On 12/13/2017 3:16 PM, Sundaram, Dinesh wrote:
> Thanks Shawn for your input, Is this errors specific only for zookeeper operations? If so is there any way to turn off default zookeeper which runs on 9983?
If you don't want to start the embedded zookeeper, then you want to be sure that you have a zkHost defined which lists all of the hosts in your external ensemble. You can either define ZK_HOST in the include script, or use the -z option when starting Solr manually. When Solr is provided with information about ZK hosts, it does NOT start the embedded ZK.
The exceptions you're seeing have nothing to do with zookeeper. The latest exception you mentioned is caused by one SolrCloud instance sending HTTPS requests to another SolrCloud instance, and failing to validate SSL because the hostname doesn't match the info in the certificate.
Thanks,
Shawn
CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Re: Solr ssl issue while creating collection
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/13/2017 3:16 PM, Sundaram, Dinesh wrote:
> Thanks Shawn for your input, Is this errors specific only for zookeeper operations? If so is there any way to turn off default zookeeper which runs on 9983?
If you don't want to start the embedded zookeeper, then you want to be
sure that you have a zkHost defined which lists all of the hosts in your
external ensemble. You can either define ZK_HOST in the include script,
or use the -z option when starting Solr manually. When Solr is provided
with information about ZK hosts, it does NOT start the embedded ZK.
The exceptions you're seeing have nothing to do with zookeeper. The
latest exception you mentioned is caused by one SolrCloud instance
sending HTTPS requests to another SolrCloud instance, and failing to
validate SSL because the hostname doesn't match the info in the certificate.
Thanks,
Shawn
RE: Solr ssl issue while creating collection
Posted by "Sundaram, Dinesh" <Di...@mastercard.com>.
Thanks Shawn for your input, Is this errors specific only for zookeeper operations? If so is there any way to turn off default zookeeper which runs on 9983?
Dinesh Sundaram
MBS Platform Engineering
Mastercard
-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Wednesday, December 13, 2017 11:38 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr ssl issue while creating collection
On 12/13/2017 10:06 AM, Sundaram, Dinesh wrote:
> Thanks Shawn, this helps. Now getting the below exception, is there any way to avoid verifying this?
>
> 2017-12-13 17:00:39.239 DEBUG (httpShardExecutor-4-thread-1-processing-n:xx.xx.xx.xx:8983_solr [https://urldefense.proofpoint.com/v2/url?u=https-3A____xx.xx.xx.xx-3A8983__solr&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=v4DznkLF4VBvrVleiFON0I41uu_NPGd1TpVYs3q0Hro&s=eqDSyAa-0UCXm_IT2YoWaZDjMb5zM5Uv8-9Zcidjlec&e=] https://urldefense.proofpoint.com/v2/url?u=https-3A____xx.xx.xx.xx-3A8983__solr&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=v4DznkLF4VBvrVleiFON0I41uu_NPGd1TpVYs3q0Hro&s=eqDSyAa-0UCXm_IT2YoWaZDjMb5zM5Uv8-9Zcidjlec&e=) [ ] o.a.h.c.s.DefaultHostnameVerifier Certificate for <xx.xx.xx.xx> doesn't match common name of the certificate subject: xx.xx.xx.xx.com
> javax.net.ssl.SSLPeerUnverifiedException: Certificate for
> <xx.xx.xx.xx> doesn't match common name of the certificate subject:
> xx.xx.xx.xx.com
If you're running 6.x, then you can disable the hostname verification. But if you're running 7.x, there's a bug that breaks it:
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D9304&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=v4DznkLF4VBvrVleiFON0I41uu_NPGd1TpVYs3q0Hro&s=mX_wS19NYYqBsWUI3qCXAXBbY-3p8Vjkzq4K3BFfgdk&e=
There's a patch on the issue, but it hasn't been tested, so I have no idea whether it works. Even if it works, the patch is incomplete because it doesn't have a test to verify the problem doesn't happen again.
An alternate idea would be to add all the possible hostnames to the certificate you're using, and make sure the trust stores are valid, so all of the cert verification will work.
Thanks,
Shawn
CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Re: Solr ssl issue while creating collection
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/13/2017 10:06 AM, Sundaram, Dinesh wrote:
> Thanks Shawn, this helps. Now getting the below exception, is there any way to avoid verifying this?
>
> 2017-12-13 17:00:39.239 DEBUG (httpShardExecutor-4-thread-1-processing-n:xx.xx.xx.xx:8983_solr [https:////xx.xx.xx.xx:8983//solr] https:////xx.xx.xx.xx:8983//solr) [ ] o.a.h.c.s.DefaultHostnameVerifier Certificate for <xx.xx.xx.xx> doesn't match common name of the certificate subject: xx.xx.xx.xx.com
> javax.net.ssl.SSLPeerUnverifiedException: Certificate for <xx.xx.xx.xx> doesn't match common name of the certificate subject: xx.xx.xx.xx.com
If you're running 6.x, then you can disable the hostname verification.
But if you're running 7.x, there's a bug that breaks it:
https://issues.apache.org/jira/browse/SOLR-9304
There's a patch on the issue, but it hasn't been tested, so I have no
idea whether it works. Even if it works, the patch is incomplete
because it doesn't have a test to verify the problem doesn't happen again.
An alternate idea would be to add all the possible hostnames to the
certificate you're using, and make sure the trust stores are valid, so
all of the cert verification will work.
Thanks,
Shawn
RE: Solr ssl issue while creating collection
Posted by "Sundaram, Dinesh" <Di...@mastercard.com>.
Thanks Shawn, this helps. Now getting the below exception, is there any way to avoid verifying this?
2017-12-13 17:00:39.239 DEBUG (httpShardExecutor-4-thread-1-processing-n:xx.xx.xx.xx:8983_solr [https:////xx.xx.xx.xx:8983//solr] https:////xx.xx.xx.xx:8983//solr) [ ] o.a.h.c.s.DefaultHostnameVerifier Certificate for <xx.xx.xx.xx> doesn't match common name of the certificate subject: xx.xx.xx.xx.com
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <xx.xx.xx.xx> doesn't match common name of the certificate subject: xx.xx.xx.xx.com
2017-12-13 17:00:39.242 ERROR (OverseerThreadFactory-8-thread-1-processing-n:xx.xx.xx.xx:8983_solr) [ ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://xx.xx.xx.xx:8983/solr
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://xx.xx.xx.xx:8983/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:640)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <xx.xx.xx.xx> doesn't match any of the subject alternative names: []
Dinesh Sundaram
MBS Platform Engineering
Mastercard
-----Original Message-----
From: Shawn Heisey [mailto:apache@elyograg.org]
Sent: Monday, December 11, 2017 2:26 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr ssl issue while creating collection
On 12/11/2017 12:24 PM, Sundaram, Dinesh wrote:
> 1. Configure SSL
> using
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org
> _solr_guide_7-5F1_enabling-2Dssl.html&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xT
> CjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m
> =kX8SMKw_W4qlgQyvl3p8pLrhYorEW4_wklVchKw6jAA&s=Gz_ER-vMMwpE5j1YpqIrjnf
> _P3SM7uPI-kpjGdeATR8&e=
>
> 2. Restart solr
> 3. Validate solr with https url
> https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A8983_
> solr&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7
> y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=kX8SMKw_W4qlgQyvl3p8pLrhYorEW4_w
> klVchKw6jAA&s=EJ68RQ28Gn6vNdedX5n0hue_hgqlEWR9jFWoEbkt7J4&e= - works
> fine 4. Create a collection
> https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A8983_
> solr_-23_-7Ecollections&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP
> 0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=kX8SMKw_W4qlg
> Qyvl3p8pLrhYorEW4_wklVchKw6jAA&s=weJY5eOZccSQqlLFr5CAH7PEyWPL1fb5VaWKG
> AjYAJs&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A8983
> _solr_-23_-257Ecollections&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF
> 6SP0bDlmMmY&r=gCFZFMR7y0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=kX8SMKw_W4
> qlgQyvl3p8pLrhYorEW4_wklVchKw6jAA&s=CdjOnW9WrZwGNv5Rr3kEke61pipUE8kMVA
> 9DzaYluRU&e=>
> 5. here is the response :
> Connection to Solr lost
> Please check the Solr instance.
> 6.Server solr.log: here notice the replica call goes to http port
> instead of https
>
> 2017-12-11 11:52:27.929 ERROR
> (OverseerThreadFactory-8-thread-1-processing-n:localhost:8983_solr) [
> ] o.a.s.c.OverseerCollectionMessageHandler Error from
> shard:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8983_s
> olr&d=DwIDaQ&c=uc5ZRXl8dGLM1RMQwf7xTCjRqXF0jmCF6SP0bDlmMmY&r=gCFZFMR7y
> 0gzhIBFz1lKTqHFMl-3R6gq7ojE0Eam2Eg&m=kX8SMKw_W4qlgQyvl3p8pLrhYorEW4_wk
> lVchKw6jAA&s=MjaZgIhWcEaKn00NFazu0zGn3HFKeSuYOlhyKe9RJMs&e=
>
This acts like either you did not set the urlScheme cluster property in zookeeper to https, or that you did not restart your Solr instances after making that change. Setting the property is described on the page you referenced in the "SSL with SolrCloud" section.
Note that it also appears your Solr instances have registered themselves with the "localhost" name instead of an actual IP address or a "real"
hostname. This is going to be a problem if you ever run more than one Solr machine in your cloud, or if you use a smart client (like CloudSolrClient included with SolrJ) and access Solr from a different machine.
Thanks,
Shawn
CONFIDENTIALITY NOTICE This e-mail message and any attachments are only for the use of the intended recipient and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient, any disclosure, distribution or other use of this e-mail message or attachments is prohibited. If you have received this e-mail message in error, please delete and notify the sender immediately. Thank you.
Re: Solr ssl issue while creating collection
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/11/2017 12:24 PM, Sundaram, Dinesh wrote:
> 1. Configure SSL
> using https://lucene.apache.org/solr/guide/7_1/enabling-ssl.html
>
> 2. Restart solr
> 3. Validate solr with https url https://localhost:8983/solr - works fine
> 4. Create a collection https://localhost:8983/solr/#/~collections
> <https://localhost:8983/solr/#/%7Ecollections>
> 5. here is the response :
> Connection to Solr lost
> Please check the Solr instance.
> 6.Server solr.log: here notice the replica call goes to http port
> instead of https
>
> 2017-12-11 11:52:27.929 ERROR
> (OverseerThreadFactory-8-thread-1-processing-n:localhost:8983_solr) [
> ] o.a.s.c.OverseerCollectionMessageHandler Error from
> shard: http://localhost:8983/solr
>
This acts like either you did not set the urlScheme cluster property in
zookeeper to https, or that you did not restart your Solr instances
after making that change. Setting the property is described on the page
you referenced in the "SSL with SolrCloud" section.
Note that it also appears your Solr instances have registered themselves
with the "localhost" name instead of an actual IP address or a "real"
hostname. This is going to be a problem if you ever run more than one
Solr machine in your cloud, or if you use a smart client (like
CloudSolrClient included with SolrJ) and access Solr from a different
machine.
Thanks,
Shawn