You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kevin Risden <kr...@apache.org> on 2020/07/13 21:16:14 UTC

Re: [CAUTION] SSL + Solr 8.5.1 in cloud mode + Java 8

>
> In local with just certificate and one domain name  the SSL communication
> worked. With multiple DNS and 2 certificates SSL fails with below exception.
>

A client keystore by definition can only have a single certificate. A
server keystore can have multiple certificates. The reason being is that a
client can only be identified by a single certificate.

Can you share more details about specifically what your solr.in.sh configs
look like related to keystore/truststore and which files? Specifically
highlight which files have multiple certificates in them.

It looks like for the Solr internal http client, the client keystore has
more than one certificate in it and the error is correct. This is more
strict with recent versions of Jetty 9.4.x. Previously this would silently
fail, but was still incorrect. Now the error is bubbled up so that there is
no silent misconfigurations.

Kevin Risden


On Mon, Jul 13, 2020 at 4:54 PM Natarajan, Rajeswari <
rajeswari.natarajan@sap.com> wrote:

> I looked at the patch mentioned in the JIRA
> https://issues.apache.org/jira/browse/SOLR-14105  reporting the below
> issue. I looked at the solr 8.5.1 code base , I see the patch is applied.
> But still seeing the same  exception with different stack trace. The
> initial excsption stacktrace was at
>
> at
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
>
>
> Now the exception we encounter is at httpsolrclient creation
>
>
>         Caused by: java.lang.RuntimeException:
> java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
> supported on Server
>               at
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>
> I commented the JIRA also. Let me know if this is still an issue.
>
> Thanks,
> Rajeswari
>
> On 7/13/20, 2:03 AM, "Natarajan, Rajeswari" <ra...@sap.com>
> wrote:
>
>     Re-sending to see if anyone encountered  had this combination and
> encountered this issue. In local with just certificate and one domain name
> the SSL communication worked. With multiple DNS and 2 certificates SSL
> fails with below exception.  Below JIRA says it is fixed for
> Http2SolrClient , wondering if this is fixed for http1 solr client as we
> pass -Dsolr.http1=true .
>
>     Thanks,
>     Rajeswari
>
>     https://issues.apache.org/jira/browse/SOLR-14105
>
>     On 7/6/20, 10:02 PM, "Natarajan, Rajeswari" <
> rajeswari.natarajan@sap.com> wrote:
>
>         Hi,
>
>         We are using Solr 8.5.1 in cloud mode  with Java 8. We are
> enabling  TLS  with http1  (as we get a warning java 8 + solr 8.5 SSL can’t
> be enabled) and we get below exception
>
>
>
>         2020-07-07 03:58:53.078 ERROR (main) [   ] o.a.s.c.SolrCore
> null:org.apache.solr.common.SolrException: Error instantiating
> shardHandlerFactory class [HttpShardHandlerFactory]:
> java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
> supported on Server
>               at
> org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56)
>               at
> org.apache.solr.core.CoreContainer.load(CoreContainer.java:647)
>               at
> org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:263)
>               at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:183)
>               at
> org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134)
>               at
> org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751)
>               at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
>               at
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
>               at
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
>               at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
>               at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
>               at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
>               at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
>               at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
>               at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
>               at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
>               at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
>               at
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>               at
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513)
>               at
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154)
>               at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173)
>               at
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447)
>               at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66)
>               at
> org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784)
>               at
> org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753)
>               at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641)
>               at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599)
>               at
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>               at org.eclipse.jetty.server.Server.start(Server.java:407)
>               at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>               at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
>               at org.eclipse.jetty.server.Server.doStart(Server.java:371)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
>               at java.security.AccessController.doPrivileged(Native Method)
>               at
> org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
>               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>               at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>               at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>               at java.lang.reflect.Method.invoke(Method.java:498)
>               at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
>               at org.eclipse.jetty.start.Main.start(Main.java:491)
>               at org.eclipse.jetty.start.Main.main(Main.java:77)
>         Caused by: java.lang.RuntimeException:
> java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
> supported on Server
>               at
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>               at
> org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:153)
>               at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:832)
>               at
> org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:321)
>               at
> org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51)
>               ... 50 more
>         Caused by: java.lang.UnsupportedOperationException:
> X509ExtendedKeyManager only supported on Server
>               at
> org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1273)
>               at
> org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1255)
>               at
> org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
>               at
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>               at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>               at
> org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
>               at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>               at
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:221)
>               ... 54 more
>
>
>         I see that there is a  below bug for this issue and is resolved.
> So I am not sure what will the cause of the issue.
>
>         https://issues.apache.org/jira/browse/SOLR-14105
>
>
>         Thanks,
>         Rajeswari
>
>
>

Re: [CAUTION] SSL + Solr 8.5.1 in cloud mode + Java 8

Posted by Kevin Risden <kr...@apache.org>.
You need to remove the references from bin/solr or bin/solr.cmd to
SOLR_SSL_CLIENT_KEY_STORE and "-Djavax.net.ssl.keyStore". This is different
from solr.in.sh.

The way the bin/solr script is written it is falling back to whatever is
provided as SOLR_SSL_KEY_STORE for the client keystore which is causing
issues.

Kevin Risden



On Wed, Jul 15, 2020 at 3:45 AM Natarajan, Rajeswari <
rajeswari.natarajan@sap.com> wrote:

> Thank you for your reply. I looked at solr.in.sh I see that
> SOLR_SSL_CLIENT_KEY_STORE  is already commented out by default. But you are
> right I looked at the running solr,  I see the option
> -Djavax.net.ssl.keyStore pointing to solr-ssl.keystore.p12 , not sure how
> it is getting that value. Let me dig more. Thanks for the pointer. Also if
> you have a pointer how it get's populated  other than
> SOLR_SSL_CLIENT_KEY_STORE config in solr.in.sh , please let me know
>
> #SOLR_SSL_CLIENT_KEY_STORE=
> #SOLR_SSL_CLIENT_KEY_STORE_PASSWORD=
> #SOLR_SSL_CLIENT_KEY_STORE_TYPE=
> #SOLR_SSL_CLIENT_TRUST_STORE=
> #SOLR_SSL_CLIENT_TRUST_STORE_PASSWORD=
> #SOLR_SSL_CLIENT_TRUST_STORE_TYPE=
>
> Yes we are not using Solr client auth.
>
> Thanks,
> Rajeswari
>
> On 7/14/20, 5:55 PM, "Kevin Risden" <kr...@apache.org> wrote:
>
>     Hmmm so I looked closer - it looks like a side effect of the default
>     passthrough of the keystore being passed to the client keystore.
>
>     https://github.com/apache/lucene-solr/blob/master/solr/bin/solr#L229
>
>     Can you remove or commout the entire SOLR_SSL_CLIENT_KEY_STORE section
> from
>     bin/solr or bin/solr.cmd depending on which version you are using? The
> key
>     being to make sure to not set "-Djavax.net.ssl.keyStore".
>
>     This assumes that you aren't using Solr client auth (which based on
> your
>     config you aren't) and you aren't trying to use Solr to connect to
> anything
>     that is secured via clientAuth (most likely you aren't).
>
>     If you can try this and report back that would be awesome. I think this
>     will fix the issue and it would be possible to make client auth opt in
>     instead of default fall back.
>     Kevin Risden
>
>
>
>     On Tue, Jul 14, 2020 at 1:46 AM Natarajan, Rajeswari <
>     rajeswari.natarajan@sap.com> wrote:
>
>     > Thank you so much for the response.  Below are the configs I have in
>     > solr.in.sh and I followed
>     > https://lucene.apache.org/solr/guide/8_5/enabling-ssl.html
> documentation
>     >
>     > # Enables HTTPS. It is implicitly true if you set
> SOLR_SSL_KEY_STORE. Use
>     > this config
>     > # to enable https module with custom jetty configuration.
>     > SOLR_SSL_ENABLED=true
>     > # Uncomment to set SSL-related system properties
>     > # Be sure to update the paths to the correct keystore for your
> environment
>     > SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12
>     > SOLR_SSL_KEY_STORE_PASSWORD=secret
>     > SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12
>     > SOLR_SSL_TRUST_STORE_PASSWORD=secret
>     > # Require clients to authenticate
>     > SOLR_SSL_NEED_CLIENT_AUTH=false
>     > # Enable clients to authenticate (but not require)
>     > SOLR_SSL_WANT_CLIENT_AUTH=false
>     > # SSL Certificates contain host/ip "peer name" information that is
>     > validated by default. Setting
>     > # this to false can be useful to disable these checks when re-using a
>     > certificate on many hosts
>     > SOLR_SSL_CHECK_PEER_NAME=true
>     >
>     > In local , with the below certificate it works
>     > ---------------------------------------
>     >
>     > keytool -list -keystore solr-ssl.keystore.p12
>     > Enter keystore password:
>     > Keystore type: PKCS12
>     > Keystore provider: SUN
>     >
>     > Your keystore contains 1 entry
>     >
>     > solr-18, Jun 26, 2020, PrivateKeyEntry,
>     > Certificate fingerprint (SHA1):
>     > AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
>     > C02W48C6HTD6:solr-8.5.1 i843100$ keytool -list -v -keystore
>     > solr-ssl.keystore.p12
>     > Enter keystore password:
>     > Keystore type: PKCS12
>     > Keystore provider: SUN
>     >
>     > Your keystore contains 1 entry
>     >
>     > Alias name: solr-18
>     > Creation date: Jun 26, 2020
>     > Entry type: PrivateKeyEntry
>     > Certificate chain length: 1
>     > Certificate[1]:
>     > Owner: CN=localhost, OU=Organizational Unit, O=Organization,
> L=Location,
>     > ST=State, C=Country
>     > Issuer: CN=localhost, OU=Organizational Unit, O=Organization,
> L=Location,
>     > ST=State, C=Country
>     > Serial number: 45a822c8
>     > Valid from: Fri Jun 26 00:13:03 PDT 2020 until: Sun Nov 10 23:13:03
> PST
>     > 2047
>     > Certificate fingerprints:
>     >          MD5:  0B:80:54:89:44:65:93:07:1F:81:88:8D:EC:BD:38:41
>     >          SHA1:
> AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
>     >          SHA256:
>     >
> 9D:65:A6:55:D7:22:B2:72:C2:20:55:66:F8:0C:9C:48:B1:F6:48:40:A4:FB:CB:26:77:DE:C4:97:34:69:25:42
>     > Signature algorithm name: SHA256withRSA
>     > Subject Public Key Algorithm: 2048-bit RSA key
>     > Version: 3
>     >
>     > Extensions:
>     >
>     > #1: ObjectId: 2.5.29.17 Criticality=false
>     > SubjectAlternativeName [
>     >   DNSName: localhost
>     >   IPAddress: 172.20.10.4
>     >   IPAddress: 127.0.0.1
>     > ]
>     >
>     > #2: ObjectId: 2.5.29.14 Criticality=false
>     > SubjectKeyIdentifier [
>     > KeyIdentifier [
>     > 0000: 1B 6F BB 65 A4 3C 6A F4   C9 05 08 89 88 0E 9E 76
> .o.e.<j........v
>     > 0010: A1 B7 28 BE                                        ..(.
>     > ]
>     >
>     > /////////////////////////////////////////////////////////////////
>     > In a cluster env , where the deployment  , keystore everything is
>     > automated  (used by  multiple teams) keystore generated is as below.
> As you
>     > can see the  keystore has 2 certificates , in which case I get the
>     > exception  below.
>     >
>     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
>     >     > supported on Server
>     >     >               at
>     >     >
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>     >     >
>     >
>     > In both cases , the config is same except the keystore certificates
> . In
>     > the JIRA (https://issues.apache.org/jira/browse/SOLR-14105) , I see
> the
>     > fix says it supports multiple DNS and multiple certificates. So I
> thought
>     > it should be ok. Please let me know .
>     >
>     > keytool -list -keystore  /etc/nginx/certs/sidecar.p12
>     > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
>     > Enter keystore password:
>     > Keystore type: PKCS12
>     > Keystore provider: SUN
>     >
>     > Your keystore contains 1 entry
>     >
>     > 1, Jul 7, 2020, PrivateKeyEntry,
>     > Certificate fingerprint (SHA1):
>     > E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
>     > bash-5.0#
>     > ————————-
>     >
>     > bash-5.0#  keytool -list -v -keystore /etc/nginx/certs/sidecar.p12
>     > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
>     > Enter keystore password:
>     > Keystore type: PKCS12
>     > Keystore provider: SUN
>     >
>     > Your keystore contains 1 entry
>     >
>     > Alias name: 1
>     > Creation date: Jul 7, 2020
>     > Entry type: PrivateKeyEntry
>     > Certificate chain length: 2
>     > Certificate[1]:
>     > Owner: OU=Cobalt, O=SAP, L=Walldorf, ST=Walldorf, C=DE
>     > Issuer: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP
>     > Ariba, ST=CA, C=US
>     > Serial number: 1000
>     > Valid from: Tue Jul 07 05:14:37 GMT 2020 until: Thu Jul 07 05:14:37
> GMT
>     > 2022
>     > Certificate fingerprints:
>     >          MD5:  C0:13:87:37:96:C2:E2:DD:B9:D7:B4:E3:6B:73:A0:EC
>     >          SHA1:
> E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
>     >          SHA256:
>     >
> 89:AB:8E:3B:D4:EC:A6:D0:0E:D7:CB:65:8C:92:13:32:F2:FD:7E:41:C9:39:F5:66:D5:7D:F1:04:13:8A:4E:92
>     > Signature algorithm name: SHA256withRSA
>     > Subject Public Key Algorithm: 2048-bit RSA key
>     > Version: 3
>     >
>     > Extensions:
>     >
>     > #1: ObjectId: 2.16.840.1.113730.1.13 Criticality=false
>     > 0000: 16 24 4F 70 65 6E 53 53   4C 20 47 65 6E 65 72 61  .$OpenSSL
> Genera
>     > 0010: 74 65 64 20 53 65 72 76   65 72 20 43 65 72 74 69  ted Server
> Certi
>     > 0020: 66 69 63 61 74 65                                  ficate
>     >
>     >
>     > #2: ObjectId: 2.5.29.35 Criticality=false
>     > AuthorityKeyIdentifier [
>     > KeyIdentifier [
>     > 0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D
> .\Br^p.......M..
>     > 0010: F3 37 2C 95                                        .7,.
>     > ]
>     > [CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA,
> C=US]
>     > SerialNumber: [    1001]
>     > ]
>     >
>     > #3: ObjectId: 2.5.29.19 Criticality=false
>     > BasicConstraints:[
>     >   CA:false
>     >   PathLen: undefined
>     > ]
>     >
>     > #4: ObjectId: 2.5.29.37 Criticality=false
>     > ExtendedKeyUsages [
>     >   serverAuth
>     > ]
>     >
>     > #5: ObjectId: 2.5.29.15 Criticality=true
>     > KeyUsage [
>     >   DigitalSignature
>     >   Key_Encipherment
>     > ]
>     >
>     > #6: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
>     > NetscapeCertType [
>     >    SSL server
>     > ]
>     >
>     > #7: ObjectId: 2.5.29.17 Criticality=false
>     > SubjectAlternativeName [
>     >   DNSName: search-solrcloud-solrcloud.service
>     >   DNSName: search-solrcloud-solrcloud.service.mu.aws.ariba.com
>     >   DNSName: *.query.mu.aws.ariba.com
>     >   DNSName: *.query
>     >   DNSName: *.service
>     >   DNSName:
>     >
> e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service.mu.aws.ariba.com
>     >   DNSName:
>     >
> e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service
>     >   DNSName: *.service.mu.aws.ariba.com
>     >   DNSName: 1.search-solrcloud-solrcloud.service.mu.aws.ariba.com
>     >   DNSName: 1.search-solrcloud-solrcloud.service
>     >   DNSName: localhost
>     >   IPAddress: 10.1.56.9
>     >   IPAddress: 10.169.50.16
>     >   IPAddress: 127.0.0.1
>     > ]
>     >
>     > #8: ObjectId: 2.5.29.14 Criticality=false
>     > SubjectKeyIdentifier [
>     > KeyIdentifier [
>     > 0000: 3F 9D 3D 24 48 1E 61 3C   BD C0 A4 07 8B 64 51 0D
> ?.=$H.a<.....dQ.
>     > 0010: A2 B2 FE 89                                        ....
>     > ]
>     > ]
>     >
>     > Certificate[2]:
>     > Owner: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP
>     > Ariba, ST=CA, C=US
>     > Issuer: CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto,
> ST=CA,
>     > C=US
>     > Serial number: 1001
>     > Valid from: Thu Apr 16 07:18:55 GMT 2020 until: Sun Apr 14 07:18:55
> GMT
>     > 2030
>     > Certificate fingerprints:
>     >          MD5:  FA:70:2F:DB:63:36:66:71:A6:7B:0F:46:F3:52:0B:3C
>     >          SHA1:
> 4F:27:D3:E3:12:24:64:18:B5:97:D0:BF:94:37:2D:5C:33:EA:1E:40
>     >          SHA256:
>     >
> 15:28:F4:DB:B3:D5:2E:21:6A:2E:56:47:E3:6B:D3:16:96:18:06:96:DA:5D:28:6B:34:CB:6D:FA:E8:FA:85:13
>     > Signature algorithm name: SHA256withRSA
>     > Subject Public Key Algorithm: 4096-bit RSA key
>     > Version: 3
>     >
>     > Extensions:
>     >
>     > #1: ObjectId: 2.5.29.35 Criticality=false
>     > AuthorityKeyIdentifier [
>     > KeyIdentifier [
>     > 0000: D8 A1 D1 11 50 8C 1C 2A   67 69 82 40 DF B5 68 6A  ....P..*gi.@
> ..hj
>     > 0010: E4 97 6E 32                                        ..n2
>     > ]
>     > ]
>     >
>     > #2: ObjectId: 2.5.29.19 Criticality=true
>     > BasicConstraints:[
>     >   CA:true
>     >   PathLen:0
>     > ]
>     >
>     > #3: ObjectId: 2.5.29.15 Criticality=true
>     > KeyUsage [
>     >   DigitalSignature
>     >   Key_CertSign
>     >   Crl_Sign
>     > ]
>     >
>     > #4: ObjectId: 2.5.29.14 Criticality=false
>     > SubjectKeyIdentifier [
>     > KeyIdentifier [
>     > 0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D
> .\Br^p.......M..
>     > 0010: F3 37 2C 95                                        .7,.
>     > ]
>     > ]
>     >
>     >
>     > Thanks,
>     > Rajeswari
>     >
>     > On 7/13/20, 2:16 PM, "Kevin Risden" <kr...@apache.org> wrote:
>     >
>     >     >
>     >     > In local with just certificate and one domain name  the SSL
>     > communication
>     >     > worked. With multiple DNS and 2 certificates SSL fails with
> below
>     > exception.
>     >     >
>     >
>     >     A client keystore by definition can only have a single
> certificate. A
>     >     server keystore can have multiple certificates. The reason being
> is
>     > that a
>     >     client can only be identified by a single certificate.
>     >
>     >     Can you share more details about specifically what your
> solr.in.sh
>     > configs
>     >     look like related to keystore/truststore and which files?
> Specifically
>     >     highlight which files have multiple certificates in them.
>     >
>     >     It looks like for the Solr internal http client, the client
> keystore
>     > has
>     >     more than one certificate in it and the error is correct. This
> is more
>     >     strict with recent versions of Jetty 9.4.x. Previously this would
>     > silently
>     >     fail, but was still incorrect. Now the error is bubbled up so
> that
>     > there is
>     >     no silent misconfigurations.
>     >
>     >     Kevin Risden
>     >
>     >
>     >     On Mon, Jul 13, 2020 at 4:54 PM Natarajan, Rajeswari <
>     >     rajeswari.natarajan@sap.com> wrote:
>     >
>     >     > I looked at the patch mentioned in the JIRA
>     >     > https://issues.apache.org/jira/browse/SOLR-14105  reporting
> the
>     > below
>     >     > issue. I looked at the solr 8.5.1 code base , I see the patch
> is
>     > applied.
>     >     > But still seeing the same  exception with different stack
> trace. The
>     >     > initial excsption stacktrace was at
>     >     >
>     >     > at
>     >     >
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
>     >     >
>     >     >
>     >     > Now the exception we encounter is at httpsolrclient creation
>     >     >
>     >     >
>     >     >         Caused by: java.lang.RuntimeException:
>     >     > java.lang.UnsupportedOperationException:
> X509ExtendedKeyManager only
>     >     > supported on Server
>     >     >               at
>     >     >
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>     >     >
>     >     > I commented the JIRA also. Let me know if this is still an
> issue.
>     >     >
>     >     > Thanks,
>     >     > Rajeswari
>     >     >
>     >     > On 7/13/20, 2:03 AM, "Natarajan, Rajeswari" <
>     > rajeswari.natarajan@sap.com>
>     >     > wrote:
>     >     >
>     >     >     Re-sending to see if anyone encountered  had this
> combination and
>     >     > encountered this issue. In local with just certificate and one
>     > domain name
>     >     > the SSL communication worked. With multiple DNS and 2
> certificates
>     > SSL
>     >     > fails with below exception.  Below JIRA says it is fixed for
>     >     > Http2SolrClient , wondering if this is fixed for http1 solr
> client
>     > as we
>     >     > pass -Dsolr.http1=true .
>     >     >
>     >     >     Thanks,
>     >     >     Rajeswari
>     >     >
>     >     >     https://issues.apache.org/jira/browse/SOLR-14105
>     >     >
>     >     >     On 7/6/20, 10:02 PM, "Natarajan, Rajeswari" <
>     >     > rajeswari.natarajan@sap.com> wrote:
>     >     >
>     >     >         Hi,
>     >     >
>     >     >         We are using Solr 8.5.1 in cloud mode  with Java 8. We
> are
>     >     > enabling  TLS  with http1  (as we get a warning java 8 + solr
> 8.5
>     > SSL can’t
>     >     > be enabled) and we get below exception
>     >     >
>     >     >
>     >     >
>     >     >         2020-07-07 03:58:53.078 ERROR (main) [   ]
> o.a.s.c.SolrCore
>     >     > null:org.apache.solr.common.SolrException: Error instantiating
>     >     > shardHandlerFactory class [HttpShardHandlerFactory]:
>     >     > java.lang.UnsupportedOperationException:
> X509ExtendedKeyManager only
>     >     > supported on Server
>     >     >               at
>     >     >
>     >
> org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56)
>     >     >               at
>     >     > org.apache.solr.core.CoreContainer.load(CoreContainer.java:647)
>     >     >               at
>     >     >
>     >
> org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:263)
>     >     >               at
>     >     >
>     >
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:183)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751)
>     >     >               at
>     >     >
>     >
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
>     >     >               at
>     >     >
>     >
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
>     >     >               at
>     >     >
>     >
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
>     >     >               at
>     >     >
>     >
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66)
>     >     >               at
>     >     > org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784)
>     >     >               at
>     >     >
> org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753)
>     >     >               at
>     > org.eclipse.jetty.util.Scanner.scan(Scanner.java:641)
>     >     >               at
>     > org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>     >     >               at
>     > org.eclipse.jetty.server.Server.start(Server.java:407)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
>     >     >               at
>     > org.eclipse.jetty.server.Server.doStart(Server.java:371)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
>     >     >               at
> java.security.AccessController.doPrivileged(Native
>     > Method)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
>     >     >               at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>     >     > Method)
>     >     >               at
>     >     >
>     >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     >     >               at
>     >     >
>     >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     >     >               at
> java.lang.reflect.Method.invoke(Method.java:498)
>     >     >               at
>     > org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
>     >     >               at
> org.eclipse.jetty.start.Main.start(Main.java:491)
>     >     >               at
> org.eclipse.jetty.start.Main.main(Main.java:77)
>     >     >         Caused by: java.lang.RuntimeException:
>     >     > java.lang.UnsupportedOperationException:
> X509ExtendedKeyManager only
>     >     > supported on Server
>     >     >               at
>     >     >
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>     >     >               at
>     >     >
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:153)
>     >     >               at
>     >     >
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:832)
>     >     >               at
>     >     >
>     >
> org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:321)
>     >     >               at
>     >     >
>     >
> org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51)
>     >     >               ... 50 more
>     >     >         Caused by: java.lang.UnsupportedOperationException:
>     >     > X509ExtendedKeyManager only supported on Server
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1273)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1255)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>     >     >               at
>     >     >
> org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
>     >     >               at
>     >     >
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >     >               at
>     >     >
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:221)
>     >     >               ... 54 more
>     >     >
>     >     >
>     >     >         I see that there is a  below bug for this issue and is
>     > resolved.
>     >     > So I am not sure what will the cause of the issue.
>     >     >
>     >     >         https://issues.apache.org/jira/browse/SOLR-14105
>     >     >
>     >     >
>     >     >         Thanks,
>     >     >         Rajeswari
>     >     >
>     >     >
>     >     >
>     >
>     >
>
>

Re: [CAUTION] SSL + Solr 8.5.1 in cloud mode + Java 8

Posted by "Natarajan, Rajeswari" <ra...@sap.com>.
Thank you for your reply. I looked at solr.in.sh I see that  SOLR_SSL_CLIENT_KEY_STORE  is already commented out by default. But you are right I looked at the running solr,  I see the option -Djavax.net.ssl.keyStore pointing to solr-ssl.keystore.p12 , not sure how it is getting that value. Let me dig more. Thanks for the pointer. Also if you have a pointer how it get's populated  other than SOLR_SSL_CLIENT_KEY_STORE config in solr.in.sh , please let me know

#SOLR_SSL_CLIENT_KEY_STORE=
#SOLR_SSL_CLIENT_KEY_STORE_PASSWORD=
#SOLR_SSL_CLIENT_KEY_STORE_TYPE=
#SOLR_SSL_CLIENT_TRUST_STORE=
#SOLR_SSL_CLIENT_TRUST_STORE_PASSWORD=
#SOLR_SSL_CLIENT_TRUST_STORE_TYPE=

Yes we are not using Solr client auth.

Thanks,
Rajeswari

On 7/14/20, 5:55 PM, "Kevin Risden" <kr...@apache.org> wrote:

    Hmmm so I looked closer - it looks like a side effect of the default
    passthrough of the keystore being passed to the client keystore.

    https://github.com/apache/lucene-solr/blob/master/solr/bin/solr#L229

    Can you remove or commout the entire SOLR_SSL_CLIENT_KEY_STORE section from
    bin/solr or bin/solr.cmd depending on which version you are using? The key
    being to make sure to not set "-Djavax.net.ssl.keyStore".

    This assumes that you aren't using Solr client auth (which based on your
    config you aren't) and you aren't trying to use Solr to connect to anything
    that is secured via clientAuth (most likely you aren't).

    If you can try this and report back that would be awesome. I think this
    will fix the issue and it would be possible to make client auth opt in
    instead of default fall back.
    Kevin Risden



    On Tue, Jul 14, 2020 at 1:46 AM Natarajan, Rajeswari <
    rajeswari.natarajan@sap.com> wrote:

    > Thank you so much for the response.  Below are the configs I have in
    > solr.in.sh and I followed
    > https://lucene.apache.org/solr/guide/8_5/enabling-ssl.html documentation
    >
    > # Enables HTTPS. It is implicitly true if you set SOLR_SSL_KEY_STORE. Use
    > this config
    > # to enable https module with custom jetty configuration.
    > SOLR_SSL_ENABLED=true
    > # Uncomment to set SSL-related system properties
    > # Be sure to update the paths to the correct keystore for your environment
    > SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12
    > SOLR_SSL_KEY_STORE_PASSWORD=secret
    > SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12
    > SOLR_SSL_TRUST_STORE_PASSWORD=secret
    > # Require clients to authenticate
    > SOLR_SSL_NEED_CLIENT_AUTH=false
    > # Enable clients to authenticate (but not require)
    > SOLR_SSL_WANT_CLIENT_AUTH=false
    > # SSL Certificates contain host/ip "peer name" information that is
    > validated by default. Setting
    > # this to false can be useful to disable these checks when re-using a
    > certificate on many hosts
    > SOLR_SSL_CHECK_PEER_NAME=true
    >
    > In local , with the below certificate it works
    > ---------------------------------------
    >
    > keytool -list -keystore solr-ssl.keystore.p12
    > Enter keystore password:
    > Keystore type: PKCS12
    > Keystore provider: SUN
    >
    > Your keystore contains 1 entry
    >
    > solr-18, Jun 26, 2020, PrivateKeyEntry,
    > Certificate fingerprint (SHA1):
    > AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
    > C02W48C6HTD6:solr-8.5.1 i843100$ keytool -list -v -keystore
    > solr-ssl.keystore.p12
    > Enter keystore password:
    > Keystore type: PKCS12
    > Keystore provider: SUN
    >
    > Your keystore contains 1 entry
    >
    > Alias name: solr-18
    > Creation date: Jun 26, 2020
    > Entry type: PrivateKeyEntry
    > Certificate chain length: 1
    > Certificate[1]:
    > Owner: CN=localhost, OU=Organizational Unit, O=Organization, L=Location,
    > ST=State, C=Country
    > Issuer: CN=localhost, OU=Organizational Unit, O=Organization, L=Location,
    > ST=State, C=Country
    > Serial number: 45a822c8
    > Valid from: Fri Jun 26 00:13:03 PDT 2020 until: Sun Nov 10 23:13:03 PST
    > 2047
    > Certificate fingerprints:
    >          MD5:  0B:80:54:89:44:65:93:07:1F:81:88:8D:EC:BD:38:41
    >          SHA1: AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
    >          SHA256:
    > 9D:65:A6:55:D7:22:B2:72:C2:20:55:66:F8:0C:9C:48:B1:F6:48:40:A4:FB:CB:26:77:DE:C4:97:34:69:25:42
    > Signature algorithm name: SHA256withRSA
    > Subject Public Key Algorithm: 2048-bit RSA key
    > Version: 3
    >
    > Extensions:
    >
    > #1: ObjectId: 2.5.29.17 Criticality=false
    > SubjectAlternativeName [
    >   DNSName: localhost
    >   IPAddress: 172.20.10.4
    >   IPAddress: 127.0.0.1
    > ]
    >
    > #2: ObjectId: 2.5.29.14 Criticality=false
    > SubjectKeyIdentifier [
    > KeyIdentifier [
    > 0000: 1B 6F BB 65 A4 3C 6A F4   C9 05 08 89 88 0E 9E 76  .o.e.<j........v
    > 0010: A1 B7 28 BE                                        ..(.
    > ]
    >
    > /////////////////////////////////////////////////////////////////
    > In a cluster env , where the deployment  , keystore everything is
    > automated  (used by  multiple teams) keystore generated is as below. As you
    > can see the  keystore has 2 certificates , in which case I get the
    > exception  below.
    >
    > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    >     > supported on Server
    >     >               at
    >     >
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
    >     >
    >
    > In both cases , the config is same except the keystore certificates . In
    > the JIRA (https://issues.apache.org/jira/browse/SOLR-14105) , I see the
    > fix says it supports multiple DNS and multiple certificates. So I thought
    > it should be ok. Please let me know .
    >
    > keytool -list -keystore  /etc/nginx/certs/sidecar.p12
    > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
    > Enter keystore password:
    > Keystore type: PKCS12
    > Keystore provider: SUN
    >
    > Your keystore contains 1 entry
    >
    > 1, Jul 7, 2020, PrivateKeyEntry,
    > Certificate fingerprint (SHA1):
    > E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
    > bash-5.0#
    > ————————-
    >
    > bash-5.0#  keytool -list -v -keystore /etc/nginx/certs/sidecar.p12
    > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
    > Enter keystore password:
    > Keystore type: PKCS12
    > Keystore provider: SUN
    >
    > Your keystore contains 1 entry
    >
    > Alias name: 1
    > Creation date: Jul 7, 2020
    > Entry type: PrivateKeyEntry
    > Certificate chain length: 2
    > Certificate[1]:
    > Owner: OU=Cobalt, O=SAP, L=Walldorf, ST=Walldorf, C=DE
    > Issuer: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP
    > Ariba, ST=CA, C=US
    > Serial number: 1000
    > Valid from: Tue Jul 07 05:14:37 GMT 2020 until: Thu Jul 07 05:14:37 GMT
    > 2022
    > Certificate fingerprints:
    >          MD5:  C0:13:87:37:96:C2:E2:DD:B9:D7:B4:E3:6B:73:A0:EC
    >          SHA1: E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
    >          SHA256:
    > 89:AB:8E:3B:D4:EC:A6:D0:0E:D7:CB:65:8C:92:13:32:F2:FD:7E:41:C9:39:F5:66:D5:7D:F1:04:13:8A:4E:92
    > Signature algorithm name: SHA256withRSA
    > Subject Public Key Algorithm: 2048-bit RSA key
    > Version: 3
    >
    > Extensions:
    >
    > #1: ObjectId: 2.16.840.1.113730.1.13 Criticality=false
    > 0000: 16 24 4F 70 65 6E 53 53   4C 20 47 65 6E 65 72 61  .$OpenSSL Genera
    > 0010: 74 65 64 20 53 65 72 76   65 72 20 43 65 72 74 69  ted Server Certi
    > 0020: 66 69 63 61 74 65                                  ficate
    >
    >
    > #2: ObjectId: 2.5.29.35 Criticality=false
    > AuthorityKeyIdentifier [
    > KeyIdentifier [
    > 0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D  .\Br^p.......M..
    > 0010: F3 37 2C 95                                        .7,.
    > ]
    > [CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA, C=US]
    > SerialNumber: [    1001]
    > ]
    >
    > #3: ObjectId: 2.5.29.19 Criticality=false
    > BasicConstraints:[
    >   CA:false
    >   PathLen: undefined
    > ]
    >
    > #4: ObjectId: 2.5.29.37 Criticality=false
    > ExtendedKeyUsages [
    >   serverAuth
    > ]
    >
    > #5: ObjectId: 2.5.29.15 Criticality=true
    > KeyUsage [
    >   DigitalSignature
    >   Key_Encipherment
    > ]
    >
    > #6: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
    > NetscapeCertType [
    >    SSL server
    > ]
    >
    > #7: ObjectId: 2.5.29.17 Criticality=false
    > SubjectAlternativeName [
    >   DNSName: search-solrcloud-solrcloud.service
    >   DNSName: search-solrcloud-solrcloud.service.mu.aws.ariba.com
    >   DNSName: *.query.mu.aws.ariba.com
    >   DNSName: *.query
    >   DNSName: *.service
    >   DNSName:
    > e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service.mu.aws.ariba.com
    >   DNSName:
    > e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service
    >   DNSName: *.service.mu.aws.ariba.com
    >   DNSName: 1.search-solrcloud-solrcloud.service.mu.aws.ariba.com
    >   DNSName: 1.search-solrcloud-solrcloud.service
    >   DNSName: localhost
    >   IPAddress: 10.1.56.9
    >   IPAddress: 10.169.50.16
    >   IPAddress: 127.0.0.1
    > ]
    >
    > #8: ObjectId: 2.5.29.14 Criticality=false
    > SubjectKeyIdentifier [
    > KeyIdentifier [
    > 0000: 3F 9D 3D 24 48 1E 61 3C   BD C0 A4 07 8B 64 51 0D  ?.=$H.a<.....dQ.
    > 0010: A2 B2 FE 89                                        ....
    > ]
    > ]
    >
    > Certificate[2]:
    > Owner: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP
    > Ariba, ST=CA, C=US
    > Issuer: CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA,
    > C=US
    > Serial number: 1001
    > Valid from: Thu Apr 16 07:18:55 GMT 2020 until: Sun Apr 14 07:18:55 GMT
    > 2030
    > Certificate fingerprints:
    >          MD5:  FA:70:2F:DB:63:36:66:71:A6:7B:0F:46:F3:52:0B:3C
    >          SHA1: 4F:27:D3:E3:12:24:64:18:B5:97:D0:BF:94:37:2D:5C:33:EA:1E:40
    >          SHA256:
    > 15:28:F4:DB:B3:D5:2E:21:6A:2E:56:47:E3:6B:D3:16:96:18:06:96:DA:5D:28:6B:34:CB:6D:FA:E8:FA:85:13
    > Signature algorithm name: SHA256withRSA
    > Subject Public Key Algorithm: 4096-bit RSA key
    > Version: 3
    >
    > Extensions:
    >
    > #1: ObjectId: 2.5.29.35 Criticality=false
    > AuthorityKeyIdentifier [
    > KeyIdentifier [
    > 0000: D8 A1 D1 11 50 8C 1C 2A   67 69 82 40 DF B5 68 6A  ....P..*gi.@..hj
    > 0010: E4 97 6E 32                                        ..n2
    > ]
    > ]
    >
    > #2: ObjectId: 2.5.29.19 Criticality=true
    > BasicConstraints:[
    >   CA:true
    >   PathLen:0
    > ]
    >
    > #3: ObjectId: 2.5.29.15 Criticality=true
    > KeyUsage [
    >   DigitalSignature
    >   Key_CertSign
    >   Crl_Sign
    > ]
    >
    > #4: ObjectId: 2.5.29.14 Criticality=false
    > SubjectKeyIdentifier [
    > KeyIdentifier [
    > 0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D  .\Br^p.......M..
    > 0010: F3 37 2C 95                                        .7,.
    > ]
    > ]
    >
    >
    > Thanks,
    > Rajeswari
    >
    > On 7/13/20, 2:16 PM, "Kevin Risden" <kr...@apache.org> wrote:
    >
    >     >
    >     > In local with just certificate and one domain name  the SSL
    > communication
    >     > worked. With multiple DNS and 2 certificates SSL fails with below
    > exception.
    >     >
    >
    >     A client keystore by definition can only have a single certificate. A
    >     server keystore can have multiple certificates. The reason being is
    > that a
    >     client can only be identified by a single certificate.
    >
    >     Can you share more details about specifically what your solr.in.sh
    > configs
    >     look like related to keystore/truststore and which files? Specifically
    >     highlight which files have multiple certificates in them.
    >
    >     It looks like for the Solr internal http client, the client keystore
    > has
    >     more than one certificate in it and the error is correct. This is more
    >     strict with recent versions of Jetty 9.4.x. Previously this would
    > silently
    >     fail, but was still incorrect. Now the error is bubbled up so that
    > there is
    >     no silent misconfigurations.
    >
    >     Kevin Risden
    >
    >
    >     On Mon, Jul 13, 2020 at 4:54 PM Natarajan, Rajeswari <
    >     rajeswari.natarajan@sap.com> wrote:
    >
    >     > I looked at the patch mentioned in the JIRA
    >     > https://issues.apache.org/jira/browse/SOLR-14105  reporting the
    > below
    >     > issue. I looked at the solr 8.5.1 code base , I see the patch is
    > applied.
    >     > But still seeing the same  exception with different stack trace. The
    >     > initial excsption stacktrace was at
    >     >
    >     > at
    >     >
    > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
    >     >
    >     >
    >     > Now the exception we encounter is at httpsolrclient creation
    >     >
    >     >
    >     >         Caused by: java.lang.RuntimeException:
    >     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    >     > supported on Server
    >     >               at
    >     >
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
    >     >
    >     > I commented the JIRA also. Let me know if this is still an issue.
    >     >
    >     > Thanks,
    >     > Rajeswari
    >     >
    >     > On 7/13/20, 2:03 AM, "Natarajan, Rajeswari" <
    > rajeswari.natarajan@sap.com>
    >     > wrote:
    >     >
    >     >     Re-sending to see if anyone encountered  had this combination and
    >     > encountered this issue. In local with just certificate and one
    > domain name
    >     > the SSL communication worked. With multiple DNS and 2 certificates
    > SSL
    >     > fails with below exception.  Below JIRA says it is fixed for
    >     > Http2SolrClient , wondering if this is fixed for http1 solr client
    > as we
    >     > pass -Dsolr.http1=true .
    >     >
    >     >     Thanks,
    >     >     Rajeswari
    >     >
    >     >     https://issues.apache.org/jira/browse/SOLR-14105
    >     >
    >     >     On 7/6/20, 10:02 PM, "Natarajan, Rajeswari" <
    >     > rajeswari.natarajan@sap.com> wrote:
    >     >
    >     >         Hi,
    >     >
    >     >         We are using Solr 8.5.1 in cloud mode  with Java 8. We are
    >     > enabling  TLS  with http1  (as we get a warning java 8 + solr 8.5
    > SSL can’t
    >     > be enabled) and we get below exception
    >     >
    >     >
    >     >
    >     >         2020-07-07 03:58:53.078 ERROR (main) [   ] o.a.s.c.SolrCore
    >     > null:org.apache.solr.common.SolrException: Error instantiating
    >     > shardHandlerFactory class [HttpShardHandlerFactory]:
    >     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    >     > supported on Server
    >     >               at
    >     >
    > org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56)
    >     >               at
    >     > org.apache.solr.core.CoreContainer.load(CoreContainer.java:647)
    >     >               at
    >     >
    > org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:263)
    >     >               at
    >     >
    > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:183)
    >     >               at
    >     >
    > org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134)
    >     >               at
    >     >
    > org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751)
    >     >               at
    >     >
    > java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    >     >               at
    >     >
    > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
    >     >               at
    >     >
    > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
    >     >               at
    >     >
    > java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
    >     >               at
    >     >
    > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
    >     >               at
    >     >
    > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
    >     >               at
    >     >
    > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
    >     >               at
    >     >
    > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
    >     >               at
    >     >
    > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
    >     >               at
    >     >
    > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
    >     >               at
    >     >
    > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66)
    >     >               at
    >     > org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784)
    >     >               at
    >     > org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753)
    >     >               at
    > org.eclipse.jetty.util.Scanner.scan(Scanner.java:641)
    >     >               at
    > org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599)
    >     >               at
    >     >
    > org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    >     >               at
    > org.eclipse.jetty.server.Server.start(Server.java:407)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    >     >               at
    >     >
    > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
    >     >               at
    > org.eclipse.jetty.server.Server.doStart(Server.java:371)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
    >     >               at java.security.AccessController.doPrivileged(Native
    > Method)
    >     >               at
    >     >
    > org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
    >     >               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
    >     > Method)
    >     >               at
    >     >
    > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    >     >               at
    >     >
    > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    >     >               at java.lang.reflect.Method.invoke(Method.java:498)
    >     >               at
    > org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
    >     >               at org.eclipse.jetty.start.Main.start(Main.java:491)
    >     >               at org.eclipse.jetty.start.Main.main(Main.java:77)
    >     >         Caused by: java.lang.RuntimeException:
    >     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    >     > supported on Server
    >     >               at
    >     >
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
    >     >               at
    >     >
    > org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:153)
    >     >               at
    >     >
    > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:832)
    >     >               at
    >     >
    > org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:321)
    >     >               at
    >     >
    > org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51)
    >     >               ... 50 more
    >     >         Caused by: java.lang.UnsupportedOperationException:
    >     > X509ExtendedKeyManager only supported on Server
    >     >               at
    >     >
    > org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1273)
    >     >               at
    >     >
    > org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1255)
    >     >               at
    >     >
    > org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
    >     >               at
    >     >
    > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    >     >               at
    >     > org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
    >     >               at
    >     >
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >     >               at
    >     >
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:221)
    >     >               ... 54 more
    >     >
    >     >
    >     >         I see that there is a  below bug for this issue and is
    > resolved.
    >     > So I am not sure what will the cause of the issue.
    >     >
    >     >         https://issues.apache.org/jira/browse/SOLR-14105
    >     >
    >     >
    >     >         Thanks,
    >     >         Rajeswari
    >     >
    >     >
    >     >
    >
    >


Re: [CAUTION] SSL + Solr 8.5.1 in cloud mode + Java 8

Posted by Kevin Risden <kr...@apache.org>.
Hmmm so I looked closer - it looks like a side effect of the default
passthrough of the keystore being passed to the client keystore.

https://github.com/apache/lucene-solr/blob/master/solr/bin/solr#L229

Can you remove or commout the entire SOLR_SSL_CLIENT_KEY_STORE section from
bin/solr or bin/solr.cmd depending on which version you are using? The key
being to make sure to not set "-Djavax.net.ssl.keyStore".

This assumes that you aren't using Solr client auth (which based on your
config you aren't) and you aren't trying to use Solr to connect to anything
that is secured via clientAuth (most likely you aren't).

If you can try this and report back that would be awesome. I think this
will fix the issue and it would be possible to make client auth opt in
instead of default fall back.
Kevin Risden



On Tue, Jul 14, 2020 at 1:46 AM Natarajan, Rajeswari <
rajeswari.natarajan@sap.com> wrote:

> Thank you so much for the response.  Below are the configs I have in
> solr.in.sh and I followed
> https://lucene.apache.org/solr/guide/8_5/enabling-ssl.html documentation
>
> # Enables HTTPS. It is implicitly true if you set SOLR_SSL_KEY_STORE. Use
> this config
> # to enable https module with custom jetty configuration.
> SOLR_SSL_ENABLED=true
> # Uncomment to set SSL-related system properties
> # Be sure to update the paths to the correct keystore for your environment
> SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12
> SOLR_SSL_KEY_STORE_PASSWORD=secret
> SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12
> SOLR_SSL_TRUST_STORE_PASSWORD=secret
> # Require clients to authenticate
> SOLR_SSL_NEED_CLIENT_AUTH=false
> # Enable clients to authenticate (but not require)
> SOLR_SSL_WANT_CLIENT_AUTH=false
> # SSL Certificates contain host/ip "peer name" information that is
> validated by default. Setting
> # this to false can be useful to disable these checks when re-using a
> certificate on many hosts
> SOLR_SSL_CHECK_PEER_NAME=true
>
> In local , with the below certificate it works
> ---------------------------------------
>
> keytool -list -keystore solr-ssl.keystore.p12
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
>
> Your keystore contains 1 entry
>
> solr-18, Jun 26, 2020, PrivateKeyEntry,
> Certificate fingerprint (SHA1):
> AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
> C02W48C6HTD6:solr-8.5.1 i843100$ keytool -list -v -keystore
> solr-ssl.keystore.p12
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
>
> Your keystore contains 1 entry
>
> Alias name: solr-18
> Creation date: Jun 26, 2020
> Entry type: PrivateKeyEntry
> Certificate chain length: 1
> Certificate[1]:
> Owner: CN=localhost, OU=Organizational Unit, O=Organization, L=Location,
> ST=State, C=Country
> Issuer: CN=localhost, OU=Organizational Unit, O=Organization, L=Location,
> ST=State, C=Country
> Serial number: 45a822c8
> Valid from: Fri Jun 26 00:13:03 PDT 2020 until: Sun Nov 10 23:13:03 PST
> 2047
> Certificate fingerprints:
>          MD5:  0B:80:54:89:44:65:93:07:1F:81:88:8D:EC:BD:38:41
>          SHA1: AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
>          SHA256:
> 9D:65:A6:55:D7:22:B2:72:C2:20:55:66:F8:0C:9C:48:B1:F6:48:40:A4:FB:CB:26:77:DE:C4:97:34:69:25:42
> Signature algorithm name: SHA256withRSA
> Subject Public Key Algorithm: 2048-bit RSA key
> Version: 3
>
> Extensions:
>
> #1: ObjectId: 2.5.29.17 Criticality=false
> SubjectAlternativeName [
>   DNSName: localhost
>   IPAddress: 172.20.10.4
>   IPAddress: 127.0.0.1
> ]
>
> #2: ObjectId: 2.5.29.14 Criticality=false
> SubjectKeyIdentifier [
> KeyIdentifier [
> 0000: 1B 6F BB 65 A4 3C 6A F4   C9 05 08 89 88 0E 9E 76  .o.e.<j........v
> 0010: A1 B7 28 BE                                        ..(.
> ]
>
> /////////////////////////////////////////////////////////////////
> In a cluster env , where the deployment  , keystore everything is
> automated  (used by  multiple teams) keystore generated is as below. As you
> can see the  keystore has 2 certificates , in which case I get the
> exception  below.
>
> java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
>     > supported on Server
>     >               at
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>     >
>
> In both cases , the config is same except the keystore certificates . In
> the JIRA (https://issues.apache.org/jira/browse/SOLR-14105) , I see the
> fix says it supports multiple DNS and multiple certificates. So I thought
> it should be ok. Please let me know .
>
> keytool -list -keystore  /etc/nginx/certs/sidecar.p12
> Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
>
> Your keystore contains 1 entry
>
> 1, Jul 7, 2020, PrivateKeyEntry,
> Certificate fingerprint (SHA1):
> E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
> bash-5.0#
> ————————-
>
> bash-5.0#  keytool -list -v -keystore /etc/nginx/certs/sidecar.p12
> Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
> Enter keystore password:
> Keystore type: PKCS12
> Keystore provider: SUN
>
> Your keystore contains 1 entry
>
> Alias name: 1
> Creation date: Jul 7, 2020
> Entry type: PrivateKeyEntry
> Certificate chain length: 2
> Certificate[1]:
> Owner: OU=Cobalt, O=SAP, L=Walldorf, ST=Walldorf, C=DE
> Issuer: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP
> Ariba, ST=CA, C=US
> Serial number: 1000
> Valid from: Tue Jul 07 05:14:37 GMT 2020 until: Thu Jul 07 05:14:37 GMT
> 2022
> Certificate fingerprints:
>          MD5:  C0:13:87:37:96:C2:E2:DD:B9:D7:B4:E3:6B:73:A0:EC
>          SHA1: E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
>          SHA256:
> 89:AB:8E:3B:D4:EC:A6:D0:0E:D7:CB:65:8C:92:13:32:F2:FD:7E:41:C9:39:F5:66:D5:7D:F1:04:13:8A:4E:92
> Signature algorithm name: SHA256withRSA
> Subject Public Key Algorithm: 2048-bit RSA key
> Version: 3
>
> Extensions:
>
> #1: ObjectId: 2.16.840.1.113730.1.13 Criticality=false
> 0000: 16 24 4F 70 65 6E 53 53   4C 20 47 65 6E 65 72 61  .$OpenSSL Genera
> 0010: 74 65 64 20 53 65 72 76   65 72 20 43 65 72 74 69  ted Server Certi
> 0020: 66 69 63 61 74 65                                  ficate
>
>
> #2: ObjectId: 2.5.29.35 Criticality=false
> AuthorityKeyIdentifier [
> KeyIdentifier [
> 0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D  .\Br^p.......M..
> 0010: F3 37 2C 95                                        .7,.
> ]
> [CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA, C=US]
> SerialNumber: [    1001]
> ]
>
> #3: ObjectId: 2.5.29.19 Criticality=false
> BasicConstraints:[
>   CA:false
>   PathLen: undefined
> ]
>
> #4: ObjectId: 2.5.29.37 Criticality=false
> ExtendedKeyUsages [
>   serverAuth
> ]
>
> #5: ObjectId: 2.5.29.15 Criticality=true
> KeyUsage [
>   DigitalSignature
>   Key_Encipherment
> ]
>
> #6: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
> NetscapeCertType [
>    SSL server
> ]
>
> #7: ObjectId: 2.5.29.17 Criticality=false
> SubjectAlternativeName [
>   DNSName: search-solrcloud-solrcloud.service
>   DNSName: search-solrcloud-solrcloud.service.mu.aws.ariba.com
>   DNSName: *.query.mu.aws.ariba.com
>   DNSName: *.query
>   DNSName: *.service
>   DNSName:
> e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service.mu.aws.ariba.com
>   DNSName:
> e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service
>   DNSName: *.service.mu.aws.ariba.com
>   DNSName: 1.search-solrcloud-solrcloud.service.mu.aws.ariba.com
>   DNSName: 1.search-solrcloud-solrcloud.service
>   DNSName: localhost
>   IPAddress: 10.1.56.9
>   IPAddress: 10.169.50.16
>   IPAddress: 127.0.0.1
> ]
>
> #8: ObjectId: 2.5.29.14 Criticality=false
> SubjectKeyIdentifier [
> KeyIdentifier [
> 0000: 3F 9D 3D 24 48 1E 61 3C   BD C0 A4 07 8B 64 51 0D  ?.=$H.a<.....dQ.
> 0010: A2 B2 FE 89                                        ....
> ]
> ]
>
> Certificate[2]:
> Owner: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP
> Ariba, ST=CA, C=US
> Issuer: CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA,
> C=US
> Serial number: 1001
> Valid from: Thu Apr 16 07:18:55 GMT 2020 until: Sun Apr 14 07:18:55 GMT
> 2030
> Certificate fingerprints:
>          MD5:  FA:70:2F:DB:63:36:66:71:A6:7B:0F:46:F3:52:0B:3C
>          SHA1: 4F:27:D3:E3:12:24:64:18:B5:97:D0:BF:94:37:2D:5C:33:EA:1E:40
>          SHA256:
> 15:28:F4:DB:B3:D5:2E:21:6A:2E:56:47:E3:6B:D3:16:96:18:06:96:DA:5D:28:6B:34:CB:6D:FA:E8:FA:85:13
> Signature algorithm name: SHA256withRSA
> Subject Public Key Algorithm: 4096-bit RSA key
> Version: 3
>
> Extensions:
>
> #1: ObjectId: 2.5.29.35 Criticality=false
> AuthorityKeyIdentifier [
> KeyIdentifier [
> 0000: D8 A1 D1 11 50 8C 1C 2A   67 69 82 40 DF B5 68 6A  ....P..*gi.@..hj
> 0010: E4 97 6E 32                                        ..n2
> ]
> ]
>
> #2: ObjectId: 2.5.29.19 Criticality=true
> BasicConstraints:[
>   CA:true
>   PathLen:0
> ]
>
> #3: ObjectId: 2.5.29.15 Criticality=true
> KeyUsage [
>   DigitalSignature
>   Key_CertSign
>   Crl_Sign
> ]
>
> #4: ObjectId: 2.5.29.14 Criticality=false
> SubjectKeyIdentifier [
> KeyIdentifier [
> 0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D  .\Br^p.......M..
> 0010: F3 37 2C 95                                        .7,.
> ]
> ]
>
>
> Thanks,
> Rajeswari
>
> On 7/13/20, 2:16 PM, "Kevin Risden" <kr...@apache.org> wrote:
>
>     >
>     > In local with just certificate and one domain name  the SSL
> communication
>     > worked. With multiple DNS and 2 certificates SSL fails with below
> exception.
>     >
>
>     A client keystore by definition can only have a single certificate. A
>     server keystore can have multiple certificates. The reason being is
> that a
>     client can only be identified by a single certificate.
>
>     Can you share more details about specifically what your solr.in.sh
> configs
>     look like related to keystore/truststore and which files? Specifically
>     highlight which files have multiple certificates in them.
>
>     It looks like for the Solr internal http client, the client keystore
> has
>     more than one certificate in it and the error is correct. This is more
>     strict with recent versions of Jetty 9.4.x. Previously this would
> silently
>     fail, but was still incorrect. Now the error is bubbled up so that
> there is
>     no silent misconfigurations.
>
>     Kevin Risden
>
>
>     On Mon, Jul 13, 2020 at 4:54 PM Natarajan, Rajeswari <
>     rajeswari.natarajan@sap.com> wrote:
>
>     > I looked at the patch mentioned in the JIRA
>     > https://issues.apache.org/jira/browse/SOLR-14105  reporting the
> below
>     > issue. I looked at the solr 8.5.1 code base , I see the patch is
> applied.
>     > But still seeing the same  exception with different stack trace. The
>     > initial excsption stacktrace was at
>     >
>     > at
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
>     >
>     >
>     > Now the exception we encounter is at httpsolrclient creation
>     >
>     >
>     >         Caused by: java.lang.RuntimeException:
>     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
>     > supported on Server
>     >               at
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>     >
>     > I commented the JIRA also. Let me know if this is still an issue.
>     >
>     > Thanks,
>     > Rajeswari
>     >
>     > On 7/13/20, 2:03 AM, "Natarajan, Rajeswari" <
> rajeswari.natarajan@sap.com>
>     > wrote:
>     >
>     >     Re-sending to see if anyone encountered  had this combination and
>     > encountered this issue. In local with just certificate and one
> domain name
>     > the SSL communication worked. With multiple DNS and 2 certificates
> SSL
>     > fails with below exception.  Below JIRA says it is fixed for
>     > Http2SolrClient , wondering if this is fixed for http1 solr client
> as we
>     > pass -Dsolr.http1=true .
>     >
>     >     Thanks,
>     >     Rajeswari
>     >
>     >     https://issues.apache.org/jira/browse/SOLR-14105
>     >
>     >     On 7/6/20, 10:02 PM, "Natarajan, Rajeswari" <
>     > rajeswari.natarajan@sap.com> wrote:
>     >
>     >         Hi,
>     >
>     >         We are using Solr 8.5.1 in cloud mode  with Java 8. We are
>     > enabling  TLS  with http1  (as we get a warning java 8 + solr 8.5
> SSL can’t
>     > be enabled) and we get below exception
>     >
>     >
>     >
>     >         2020-07-07 03:58:53.078 ERROR (main) [   ] o.a.s.c.SolrCore
>     > null:org.apache.solr.common.SolrException: Error instantiating
>     > shardHandlerFactory class [HttpShardHandlerFactory]:
>     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
>     > supported on Server
>     >               at
>     >
> org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56)
>     >               at
>     > org.apache.solr.core.CoreContainer.load(CoreContainer.java:647)
>     >               at
>     >
> org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:263)
>     >               at
>     >
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:183)
>     >               at
>     >
> org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134)
>     >               at
>     >
> org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751)
>     >               at
>     >
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
>     >               at
>     >
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
>     >               at
>     >
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
>     >               at
>     >
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
>     >               at
>     >
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
>     >               at
>     >
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
>     >               at
>     >
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
>     >               at
>     >
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
>     >               at
>     >
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
>     >               at
>     >
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
>     >               at
>     >
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
>     >               at
>     >
> org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>     >               at
>     >
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513)
>     >               at
>     >
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154)
>     >               at
>     >
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173)
>     >               at
>     >
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447)
>     >               at
>     >
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66)
>     >               at
>     > org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784)
>     >               at
>     > org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753)
>     >               at
> org.eclipse.jetty.util.Scanner.scan(Scanner.java:641)
>     >               at
> org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599)
>     >               at
>     >
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>     >               at
> org.eclipse.jetty.server.Server.start(Server.java:407)
>     >               at
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>     >               at
>     >
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
>     >               at
> org.eclipse.jetty.server.Server.doStart(Server.java:371)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
>     >               at java.security.AccessController.doPrivileged(Native
> Method)
>     >               at
>     >
> org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
>     >               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>     > Method)
>     >               at
>     >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     >               at
>     >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     >               at java.lang.reflect.Method.invoke(Method.java:498)
>     >               at
> org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
>     >               at org.eclipse.jetty.start.Main.start(Main.java:491)
>     >               at org.eclipse.jetty.start.Main.main(Main.java:77)
>     >         Caused by: java.lang.RuntimeException:
>     > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
>     > supported on Server
>     >               at
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
>     >               at
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:153)
>     >               at
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:832)
>     >               at
>     >
> org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:321)
>     >               at
>     >
> org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51)
>     >               ... 50 more
>     >         Caused by: java.lang.UnsupportedOperationException:
>     > X509ExtendedKeyManager only supported on Server
>     >               at
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1273)
>     >               at
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1255)
>     >               at
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
>     >               at
>     >
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
>     >               at
>     >
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>     >               at
>     > org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
>     >               at
>     >
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
>     >               at
>     >
> org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:221)
>     >               ... 54 more
>     >
>     >
>     >         I see that there is a  below bug for this issue and is
> resolved.
>     > So I am not sure what will the cause of the issue.
>     >
>     >         https://issues.apache.org/jira/browse/SOLR-14105
>     >
>     >
>     >         Thanks,
>     >         Rajeswari
>     >
>     >
>     >
>
>

Re: [CAUTION] SSL + Solr 8.5.1 in cloud mode + Java 8

Posted by "Natarajan, Rajeswari" <ra...@sap.com>.
Thank you so much for the response.  Below are the configs I have in solr.in.sh and I followed  https://lucene.apache.org/solr/guide/8_5/enabling-ssl.html documentation

# Enables HTTPS. It is implicitly true if you set SOLR_SSL_KEY_STORE. Use this config
# to enable https module with custom jetty configuration.
SOLR_SSL_ENABLED=true
# Uncomment to set SSL-related system properties
# Be sure to update the paths to the correct keystore for your environment
SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.p12
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.p12
SOLR_SSL_TRUST_STORE_PASSWORD=secret
# Require clients to authenticate
SOLR_SSL_NEED_CLIENT_AUTH=false
# Enable clients to authenticate (but not require)
SOLR_SSL_WANT_CLIENT_AUTH=false
# SSL Certificates contain host/ip "peer name" information that is validated by default. Setting
# this to false can be useful to disable these checks when re-using a certificate on many hosts
SOLR_SSL_CHECK_PEER_NAME=true

In local , with the below certificate it works
---------------------------------------

keytool -list -keystore solr-ssl.keystore.p12
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

solr-18, Jun 26, 2020, PrivateKeyEntry, 
Certificate fingerprint (SHA1): AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
C02W48C6HTD6:solr-8.5.1 i843100$ keytool -list -v -keystore solr-ssl.keystore.p12
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: solr-18
Creation date: Jun 26, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
Issuer: CN=localhost, OU=Organizational Unit, O=Organization, L=Location, ST=State, C=Country
Serial number: 45a822c8
Valid from: Fri Jun 26 00:13:03 PDT 2020 until: Sun Nov 10 23:13:03 PST 2047
Certificate fingerprints:
	 MD5:  0B:80:54:89:44:65:93:07:1F:81:88:8D:EC:BD:38:41
	 SHA1: AB:F2:C8:84:E8:E7:A2:BF:2D:0D:2F:D3:95:4A:98:5B:2A:88:81:50
	 SHA256: 9D:65:A6:55:D7:22:B2:72:C2:20:55:66:F8:0C:9C:48:B1:F6:48:40:A4:FB:CB:26:77:DE:C4:97:34:69:25:42
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions: 

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: localhost
  IPAddress: 172.20.10.4
  IPAddress: 127.0.0.1
]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 1B 6F BB 65 A4 3C 6A F4   C9 05 08 89 88 0E 9E 76  .o.e.<j........v
0010: A1 B7 28 BE                                        ..(.
]

/////////////////////////////////////////////////////////////////
In a cluster env , where the deployment  , keystore everything is automated  (used by  multiple teams) keystore generated is as below. As you can see the  keystore has 2 certificates , in which case I get the exception  below.

java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    > supported on Server
    >               at
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
    >

In both cases , the config is same except the keystore certificates . In the JIRA (https://issues.apache.org/jira/browse/SOLR-14105) , I see the fix says it supports multiple DNS and multiple certificates. So I thought it should be ok. Please let me know .

keytool -list -keystore  /etc/nginx/certs/sidecar.p12 
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

1, Jul 7, 2020, PrivateKeyEntry, 
Certificate fingerprint (SHA1): E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
bash-5.0# 
————————-

bash-5.0#  keytool -list -v -keystore /etc/nginx/certs/sidecar.p12 
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: 1
Creation date: Jul 7, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: OU=Cobalt, O=SAP, L=Walldorf, ST=Walldorf, C=DE
Issuer: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP Ariba, ST=CA, C=US
Serial number: 1000
Valid from: Tue Jul 07 05:14:37 GMT 2020 until: Thu Jul 07 05:14:37 GMT 2022
Certificate fingerprints:
	 MD5:  C0:13:87:37:96:C2:E2:DD:B9:D7:B4:E3:6B:73:A0:EC
	 SHA1: E2:3B:4B:4A:0E:05:CF:DA:59:09:55:8D:4E:6D:8A:1D:4E:DD:D4:62
	 SHA256: 89:AB:8E:3B:D4:EC:A6:D0:0E:D7:CB:65:8C:92:13:32:F2:FD:7E:41:C9:39:F5:66:D5:7D:F1:04:13:8A:4E:92
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions: 

#1: ObjectId: 2.16.840.1.113730.1.13 Criticality=false
0000: 16 24 4F 70 65 6E 53 53   4C 20 47 65 6E 65 72 61  .$OpenSSL Genera
0010: 74 65 64 20 53 65 72 76   65 72 20 43 65 72 74 69  ted Server Certi
0020: 66 69 63 61 74 65                                  ficate


#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D  .\Br^p.......M..
0010: F3 37 2C 95                                        .7,.
]
[CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA, C=US]
SerialNumber: [    1001]
]

#3: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:false
  PathLen: undefined
]

#4: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
]

#5: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

#6: ObjectId: 2.16.840.1.113730.1.1 Criticality=false
NetscapeCertType [
   SSL server
]

#7: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: search-solrcloud-solrcloud.service
  DNSName: search-solrcloud-solrcloud.service.mu.aws.ariba.com
  DNSName: *.query.mu.aws.ariba.com
  DNSName: *.query
  DNSName: *.service
  DNSName: e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service.mu.aws.ariba.com
  DNSName: e046469b-1bb0-55f6-913f-bd6d52b238a8.search-solrcloud-solrcloud.service
  DNSName: *.service.mu.aws.ariba.com
  DNSName: 1.search-solrcloud-solrcloud.service.mu.aws.ariba.com
  DNSName: 1.search-solrcloud-solrcloud.service
  DNSName: localhost
  IPAddress: 10.1.56.9
  IPAddress: 10.169.50.16
  IPAddress: 127.0.0.1
]

#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 3F 9D 3D 24 48 1E 61 3C   BD C0 A4 07 8B 64 51 0D  ?.=$H.a<.....dQ.
0010: A2 B2 FE 89                                        ....
]
]

Certificate[2]:
Owner: CN=SAP Ariba Cobalt Sidecar Intermediate CA, OU=COBALT, O=SAP Ariba, ST=CA, C=US
Issuer: CN=SAP Ariba Cobalt CA, OU=ES, O=SAP Ariba, L=Palo Alto, ST=CA, C=US
Serial number: 1001
Valid from: Thu Apr 16 07:18:55 GMT 2020 until: Sun Apr 14 07:18:55 GMT 2030
Certificate fingerprints:
	 MD5:  FA:70:2F:DB:63:36:66:71:A6:7B:0F:46:F3:52:0B:3C
	 SHA1: 4F:27:D3:E3:12:24:64:18:B5:97:D0:BF:94:37:2D:5C:33:EA:1E:40
	 SHA256: 15:28:F4:DB:B3:D5:2E:21:6A:2E:56:47:E3:6B:D3:16:96:18:06:96:DA:5D:28:6B:34:CB:6D:FA:E8:FA:85:13
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: D8 A1 D1 11 50 8C 1C 2A   67 69 82 40 DF B5 68 6A  ....P..*gi.@..hj
0010: E4 97 6E 32                                        ..n2
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: E9 5C 42 72 5E 70 D9 02   05 AA 11 BA 0D 4D 8D 0D  .\Br^p.......M..
0010: F3 37 2C 95                                        .7,.
]
]


Thanks,
Rajeswari

On 7/13/20, 2:16 PM, "Kevin Risden" <kr...@apache.org> wrote:

    >
    > In local with just certificate and one domain name  the SSL communication
    > worked. With multiple DNS and 2 certificates SSL fails with below exception.
    >

    A client keystore by definition can only have a single certificate. A
    server keystore can have multiple certificates. The reason being is that a
    client can only be identified by a single certificate.

    Can you share more details about specifically what your solr.in.sh configs
    look like related to keystore/truststore and which files? Specifically
    highlight which files have multiple certificates in them.

    It looks like for the Solr internal http client, the client keystore has
    more than one certificate in it and the error is correct. This is more
    strict with recent versions of Jetty 9.4.x. Previously this would silently
    fail, but was still incorrect. Now the error is bubbled up so that there is
    no silent misconfigurations.

    Kevin Risden


    On Mon, Jul 13, 2020 at 4:54 PM Natarajan, Rajeswari <
    rajeswari.natarajan@sap.com> wrote:

    > I looked at the patch mentioned in the JIRA
    > https://issues.apache.org/jira/browse/SOLR-14105  reporting the below
    > issue. I looked at the solr 8.5.1 code base , I see the patch is applied.
    > But still seeing the same  exception with different stack trace. The
    > initial excsption stacktrace was at
    >
    > at
    > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
    >
    >
    > Now the exception we encounter is at httpsolrclient creation
    >
    >
    >         Caused by: java.lang.RuntimeException:
    > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    > supported on Server
    >               at
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
    >
    > I commented the JIRA also. Let me know if this is still an issue.
    >
    > Thanks,
    > Rajeswari
    >
    > On 7/13/20, 2:03 AM, "Natarajan, Rajeswari" <ra...@sap.com>
    > wrote:
    >
    >     Re-sending to see if anyone encountered  had this combination and
    > encountered this issue. In local with just certificate and one domain name
    > the SSL communication worked. With multiple DNS and 2 certificates SSL
    > fails with below exception.  Below JIRA says it is fixed for
    > Http2SolrClient , wondering if this is fixed for http1 solr client as we
    > pass -Dsolr.http1=true .
    >
    >     Thanks,
    >     Rajeswari
    >
    >     https://issues.apache.org/jira/browse/SOLR-14105
    >
    >     On 7/6/20, 10:02 PM, "Natarajan, Rajeswari" <
    > rajeswari.natarajan@sap.com> wrote:
    >
    >         Hi,
    >
    >         We are using Solr 8.5.1 in cloud mode  with Java 8. We are
    > enabling  TLS  with http1  (as we get a warning java 8 + solr 8.5 SSL can’t
    > be enabled) and we get below exception
    >
    >
    >
    >         2020-07-07 03:58:53.078 ERROR (main) [   ] o.a.s.c.SolrCore
    > null:org.apache.solr.common.SolrException: Error instantiating
    > shardHandlerFactory class [HttpShardHandlerFactory]:
    > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    > supported on Server
    >               at
    > org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:56)
    >               at
    > org.apache.solr.core.CoreContainer.load(CoreContainer.java:647)
    >               at
    > org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:263)
    >               at
    > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:183)
    >               at
    > org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:134)
    >               at
    > org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:751)
    >               at
    > java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    >               at
    > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
    >               at
    > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
    >               at
    > java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
    >               at
    > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
    >               at
    > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
    >               at
    > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
    >               at
    > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
    >               at
    > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:822)
    >               at
    > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
    >               at
    > org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
    >               at
    > org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
    >               at
    > org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513)
    >               at
    > org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154)
    >               at
    > org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173)
    >               at
    > org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447)
    >               at
    > org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66)
    >               at
    > org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784)
    >               at
    > org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753)
    >               at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641)
    >               at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599)
    >               at
    > org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    >               at org.eclipse.jetty.server.Server.start(Server.java:407)
    >               at
    > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    >               at
    > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
    >               at org.eclipse.jetty.server.Server.doStart(Server.java:371)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
    >               at java.security.AccessController.doPrivileged(Native Method)
    >               at
    > org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
    >               at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
    > Method)
    >               at
    > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    >               at
    > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    >               at java.lang.reflect.Method.invoke(Method.java:498)
    >               at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
    >               at org.eclipse.jetty.start.Main.start(Main.java:491)
    >               at org.eclipse.jetty.start.Main.main(Main.java:77)
    >         Caused by: java.lang.RuntimeException:
    > java.lang.UnsupportedOperationException: X509ExtendedKeyManager only
    > supported on Server
    >               at
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:223)
    >               at
    > org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:153)
    >               at
    > org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:832)
    >               at
    > org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:321)
    >               at
    > org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:51)
    >               ... 50 more
    >         Caused by: java.lang.UnsupportedOperationException:
    > X509ExtendedKeyManager only supported on Server
    >               at
    > org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1273)
    >               at
    > org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1255)
    >               at
    > org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374)
    >               at
    > org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    >               at
    > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
    >               at
    > org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:244)
    >               at
    > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    >               at
    > org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:221)
    >               ... 54 more
    >
    >
    >         I see that there is a  below bug for this issue and is resolved.
    > So I am not sure what will the cause of the issue.
    >
    >         https://issues.apache.org/jira/browse/SOLR-14105
    >
    >
    >         Thanks,
    >         Rajeswari
    >
    >
    >