You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Victor Kretzer <Vi...@gdcit.com> on 2020/09/03 19:02:56 UTC

SolrCloud (6.6.6) SSL Setup - Unable to create collection

BACKGROUND: I'm attempting to setup SolrCloud (Solr 6.6.6) with an external zookeeper ensemble on Azure. I have three dedicated vms for the zookeeper ensemble and two for solr all running Ubuntu 18.04 LTS. I'm new to Solr (and Linux) and have been heavily relying on the Solr Ref Guide 6.6, most recently the following section on enabling ssl:



https://lucene.apache.org/solr/guide/6_6/enabling-ssl.html



So far I have:

Installed and setup zookeeper

Installed Solr (using install_solr_service.sh script) on both vms.

Followed the steps under Basic SSL Setup, generating certificates on each of the nodes.

Set the cluster-wide property to https per the Configure Zookeeper section of SolrCloud in the document

Started both nodes and have been able to navigate to them in my browser with https



If I do bin/solr status I get:



Solr process 13106 running on port 8983

{

  "solr_home":"/opt/solr-6.6.6/cloud/test2",

  "version":"6.6.6 68fa249034ba8b273955f20097700dc2fbb7a800 - ishan - 2019-03-29 09:13:13",

  "startTime":"2020-09-03T18:15:34.092Z",

  "uptime":"0 days, 0 hours, 43 minutes, 29 seconds",

  "memory":"52.7 MB (%10.7) of 490.7 MB",

  "cloud":{

    "ZooKeeper":"zk1:2181,zk2:2181,zk3:2181/solr",

    "liveNodes":"2",

    "collections":"0"}}







THE ISSUE

When I try to create a collection using the steps outlined in the above document, I get the following error:



azureuser@solr-node-01-test:/opt/solr$ sudo bin/solr create -c mycollection -shards 2 -force



Connecting to ZooKeeper at zk1:2181,zk2:2181,zk3:2181/solr ...

INFO  - 2020-09-03 18:21:26.784; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at zk1:2181,zk2:2181,zk3:2181/solr ready

Re-using existing configuration directory mycollection



Creating new collection 'mycollection' using command:

https://Solr1:8983/solr/admin/collections?action=CREATE&name=mycollection&numShards=2&replicationFactor=1&maxShardsPerNode=1&collection.configName=mycollection



ERROR: Failed to create collection 'mycollection' due to: {Solr2:8983_solr=org.apache.solr.client.solrj.SolrServerException:IOException occured when talking to server at: https://Solr2:8983/solr}

*I've attached logs at the bottom of this email.



QUESTIONS:

What am I doing wrong and how can I fix it?

Was I right to create separate certificates on each of the nodes (one cert on vm1, another cert on vm 2)?

Do I need to copy the certs for each node into the other (if so how)?



CONCLUSION

Thank you so much in advance and if there's any other information you need please let me know.

Victor

2020-09-03 18:15:35.240 INFO  (zkCallback-5-thread-1-processing-n:Solr1:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2020-09-03 18:15:40.124 INFO  (qtp401424608-45) [   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
2020-09-03 18:15:40.124 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1599156956818} status=0 QTime=23
2020-09-03 18:15:40.134 INFO  (qtp401424608-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} status=0 QTime=29
2020-09-03 18:15:40.171 INFO  (qtp401424608-13) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1599156956818 and sendToOCPQueue=true
2020-09-03 18:15:40.172 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1599156956818} status=0 QTime=1
2020-09-03 18:15:40.174 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} status=0 QTime=8
2020-09-03 18:15:58.225 INFO  (qtp401424608-14) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1599156974989} status=0 QTime=0
2020-09-03 18:15:58.231 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} status=0 QTime=7
2020-09-03 18:15:58.258 INFO  (qtp401424608-20) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1599156974989 and sendToOCPQueue=true
2020-09-03 18:15:58.258 INFO  (qtp401424608-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1599156974989} status=0 QTime=0
2020-09-03 18:15:58.263 INFO  (qtp401424608-21) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} status=0 QTime=7
2020-09-03 18:19:38.661 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=6
2020-09-03 18:19:38.687 INFO  (qtp401424608-45) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true
2020-09-03 18:19:38.691 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=4
2020-09-03 18:21:26.640 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=7
2020-09-03 18:21:26.677 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=5
2020-09-03 18:21:26.685 INFO  (qtp401424608-45) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true
2020-09-03 18:21:26.688 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=2
2020-09-03 18:21:26.799 INFO  (qtp401424608-16) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=list&wt=json and sendToOCPQueue=true
2020-09-03 18:21:26.799 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=list&wt=json} status=0 QTime=0
2020-09-03 18:21:26.809 INFO  (qtp401424608-16) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&maxShardsPerNode=1&collection.configName=mycollection&name=mycollection&action=CREATE&numShards=2&wt=json and sendToOCPQueue=true
2020-09-03 18:21:26.835 INFO  (OverseerThreadFactory-6-thread-1-processing-n:Solr1:8983_solr) [   ] o.a.s.c.CreateCollectionCmd Create collection mycollection
2020-09-03 18:21:27.068 INFO  (qtp401424608-14) [   ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=mycollection&newCollection=true&name=mycollection_shard2_replica1&action=CREATE&numShards=2&collection=mycollection&shard=shard2&wt=javabin&version=2
2020-09-03 18:21:27.104 ERROR (OverseerThreadFactory-6-thread-1-processing-n:Solr1:8983_solr) [   ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://Solr2:8983/solr
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://Solr2:8983/solr
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:164)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:1010)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1079)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 12 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:380)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:285)
        at sun.security.validator.Validator.validate(Validator.java:262)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1670)
        ... 29 more
Caused by: java.security.cert.CertPathValidatorException: signature check failed
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:375)
        ... 35 more
Caused by: java.security.SignatureException: Signature does not match.
        at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:457)
        at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
        ... 40 more


RE: SolrCloud (6.6.6) SSL Setup - Unable to create collection

Posted by Victor Kretzer <Vi...@gdcit.com>.
I solved my problem by using just the certificate from my first node and copying that to the second node. I'm not sure whether all three are necessary, but I copied: 
*	solr-ssl.keystore.jks
*	solr-ssl-keystore.p12
*	solr-ssl.pem.
If you originally made separate certificates for each node, make sure that on the additional nodes you remove those cert files before adding the files from the first node. I moved mine to a backup folder I created because I wasn't sure what I was trying would work but I think that was unnecessary.

Victor 

-----Original Message-----
From: Victor Kretzer <Vi...@gdcit.com> 
Sent: Thursday, September 3, 2020 3:03 PM
To: solr-user@lucene.apache.org
Subject: SolrCloud (6.6.6) SSL Setup - Unable to create collection

BACKGROUND: I'm attempting to setup SolrCloud (Solr 6.6.6) with an external zookeeper ensemble on Azure. I have three dedicated vms for the zookeeper ensemble and two for solr all running Ubuntu 18.04 LTS. I'm new to Solr (and Linux) and have been heavily relying on the Solr Ref Guide 6.6, most recently the following section on enabling ssl:



https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucene.apache.org%2Fsolr%2Fguide%2F6_6%2Fenabling-ssl.html&amp;data=02%7C01%7CVictorKretzer%40gdcit.com%7Ca124b9385d4a4eab744408d8503c0457%7C87b66f08478c40adbb095e93796da295%7C1%7C0%7C637347565991726245&amp;sdata=BE%2BPvrXsVzAR67Aoe3D%2FxMruuRlY2Img4aBHeuKpJY8%3D&amp;reserved=0



So far I have:

Installed and setup zookeeper

Installed Solr (using install_solr_service.sh script) on both vms.

Followed the steps under Basic SSL Setup, generating certificates on each of the nodes.

Set the cluster-wide property to https per the Configure Zookeeper section of SolrCloud in the document

Started both nodes and have been able to navigate to them in my browser with https



If I do bin/solr status I get:



Solr process 13106 running on port 8983

{

  "solr_home":"/opt/solr-6.6.6/cloud/test2",

  "version":"6.6.6 68fa249034ba8b273955f20097700dc2fbb7a800 - ishan - 2019-03-29 09:13:13",

  "startTime":"2020-09-03T18:15:34.092Z",

  "uptime":"0 days, 0 hours, 43 minutes, 29 seconds",

  "memory":"52.7 MB (%10.7) of 490.7 MB",

  "cloud":{

    "ZooKeeper":"zk1:2181,zk2:2181,zk3:2181/solr",

    "liveNodes":"2",

    "collections":"0"}}







THE ISSUE

When I try to create a collection using the steps outlined in the above document, I get the following error:



azureuser@solr-node-01-test:/opt/solr$ sudo bin/solr create -c mycollection -shards 2 -force



Connecting to ZooKeeper at zk1:2181,zk2:2181,zk3:2181/solr ...

INFO  - 2020-09-03 18:21:26.784; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at zk1:2181,zk2:2181,zk3:2181/solr ready

Re-using existing configuration directory mycollection



Creating new collection 'mycollection' using command:

https://Solr1:8983/solr/admin/collections?action=CREATE&name=mycollection&numShards=2&replicationFactor=1&maxShardsPerNode=1&collection.configName=mycollection



ERROR: Failed to create collection 'mycollection' due to: {Solr2:8983_solr=org.apache.solr.client.solrj.SolrServerException:IOException occured when talking to server at: https://Solr2:8983/solr}

*I've attached logs at the bottom of this email.



QUESTIONS:

What am I doing wrong and how can I fix it?

Was I right to create separate certificates on each of the nodes (one cert on vm1, another cert on vm 2)?

Do I need to copy the certs for each node into the other (if so how)?



CONCLUSION

Thank you so much in advance and if there's any other information you need please let me know.

Victor

2020-09-03 18:15:35.240 INFO  (zkCallback-5-thread-1-processing-n:Solr1:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2020-09-03 18:15:40.124 INFO  (qtp401424608-45) [   ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
2020-09-03 18:15:40.124 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1599156956818} status=0 QTime=23
2020-09-03 18:15:40.134 INFO  (qtp401424608-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} status=0 QTime=29
2020-09-03 18:15:40.171 INFO  (qtp401424608-13) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1599156956818 and sendToOCPQueue=true
2020-09-03 18:15:40.172 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1599156956818} status=0 QTime=1
2020-09-03 18:15:40.174 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156956818} status=0 QTime=8
2020-09-03 18:15:58.225 INFO  (qtp401424608-14) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={indexInfo=false&wt=json&_=1599156974989} status=0 QTime=0
2020-09-03 18:15:58.231 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} status=0 QTime=7
2020-09-03 18:15:58.258 INFO  (qtp401424608-20) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=json&_=1599156974989 and sendToOCPQueue=true
2020-09-03 18:15:58.258 INFO  (qtp401424608-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=json&_=1599156974989} status=0 QTime=0
2020-09-03 18:15:58.263 INFO  (qtp401424608-21) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json&_=1599156974989} status=0 QTime=7
2020-09-03 18:19:38.661 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=6
2020-09-03 18:19:38.687 INFO  (qtp401424608-45) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true
2020-09-03 18:19:38.691 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=4
2020-09-03 18:21:26.640 INFO  (qtp401424608-13) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=7
2020-09-03 18:21:26.677 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=5
2020-09-03 18:21:26.685 INFO  (qtp401424608-45) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=CLUSTERSTATUS&wt=json and sendToOCPQueue=true
2020-09-03 18:21:26.688 INFO  (qtp401424608-45) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=CLUSTERSTATUS&wt=json} status=0 QTime=2
2020-09-03 18:21:26.799 INFO  (qtp401424608-16) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=list&wt=json and sendToOCPQueue=true
2020-09-03 18:21:26.799 INFO  (qtp401424608-16) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=list&wt=json} status=0 QTime=0
2020-09-03 18:21:26.809 INFO  (qtp401424608-16) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&maxShardsPerNode=1&collection.configName=mycollection&name=mycollection&action=CREATE&numShards=2&wt=json and sendToOCPQueue=true
2020-09-03 18:21:26.835 INFO  (OverseerThreadFactory-6-thread-1-processing-n:Solr1:8983_solr) [   ] o.a.s.c.CreateCollectionCmd Create collection mycollection
2020-09-03 18:21:27.068 INFO  (qtp401424608-14) [   ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=mycollection&newCollection=true&name=mycollection_shard2_replica1&action=CREATE&numShards=2&collection=mycollection&shard=shard2&wt=javabin&version=2
2020-09-03 18:21:27.104 ERROR (OverseerThreadFactory-6-thread-1-processing-n:Solr1:8983_solr) [   ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: https://Solr2:8983/solr
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://Solr2:8983/solr
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:626)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:164)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:1010)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1079)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
... 12 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:380)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:285)
        at sun.security.validator.Validator.validate(Validator.java:262)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1670)
        ... 29 more
Caused by: java.security.cert.CertPathValidatorException: signature check failed
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:375)
        ... 35 more
Caused by: java.security.SignatureException: Signature does not match.
        at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:457)
        at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
        ... 40 more