You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Peter Moberg <pe...@gmail.com> on 2019/10/17 03:37:51 UTC

ElasticSearchClientServiceImpl not working for secured ElasticSearch

I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.

I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.

I check the nifi-app.log and it says:
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I am a bit surprised that I am not able to verify the same server certificate in the two different flows.

Completely stuck on this so if anyone have any pointers please let me know.

Thanks,

Peter

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
Peter,

Looks like this won't make it into 1.10, but it's a simple patch to apply
if you download the source code for 1.10 and apply it manually.

Mike

On Tue, Oct 22, 2019 at 2:46 PM Matt Burgess <ma...@gmail.com> wrote:

> For saving space (if you want to run from the assembly/target directory)
> you can add a -Ddir-only and it won’t build the tar.gz.
>
>
>
>
> Sent from my iPhone
>
> On Oct 22, 2019, at 2:25 PM, Mike Thomsen <mi...@gmail.com> wrote:
>
> 
> If you run `mvn clean install -DskipTests=true` from the root of the
> source folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd
> recommend testing against that as it'll be faster.
>
> On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg <pe...@gmail.com>
> wrote:
>
>> Mike,
>>
>> thanks for putting together that PR. I have built everything successfully
>> but I haven't been able to test this yet since I haven't built the new
>> Docker image.
>>
>> I assume you guys just use the ‘dockermaven’  folder to build the
>> Dockerfile with src artifacts?
>>
>> My current dev machine is running out of disk space so I am spending some
>> time trying to get it back to a runnable state. Hopefully I can build the
>> PR and test it in Kubernetes this week sometime.
>>
>> Thanks,
>>
>> Peter
>> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mi...@gmail.com>,
>> wrote:
>>
>> As a compromise, I upgraded to the latest 5.X client and manually
>> incremented Apache HttpClient to 4.5.10. PR is here:
>>
>> https://github.com/apache/nifi/pull/3828
>>
>> There are integration tests for that package that automatically startup
>> and provision an ES node, and they all passed with this configuration.
>> Hopefully this PR will fix your issue since it appears to be an external
>> dependency causing it.
>>
>> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> There's no hard and fast reason to stay with 5.X there, so you can build
>>> your own copy of 1.9.2 with that dependency upgraded if you want to try it
>>> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT.
>>>
>>> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>>> The certs in the TrustStore are marked as trusted.
>>>>
>>>> The host name specified in the ClientServiceImpl is:
>>>> https://quickstart-es-http.es-cluster:9200
>>>>
>>>> The CN field of the server certificate is:
>>>> https://quickstart-es-http.es-cluster.es.local
>>>>
>>>> So at first glance it looks like the issue would be that the CN field
>>>> differs from the host name. However, Im not getting an error message saying
>>>> that the HostName doesn’t match the CN. According to the RFC spec first
>>>> mentioned by Andy the CN field should only be used if the
>>>> SubjectiveAlternativeName is empty.
>>>>
>>>> The Server Cert has the SAN set to:  quickstart-es-http.es-cluster
>>>>
>>>> What is really strange to me is why I can connect using the
>>>> QueryElasticSearch processor with the same SSLContextService and hostname
>>>> but when I use the ClientServicesImpl it will not work.
>>>>
>>>> So I did some more investigation and it looks like the QueryES
>>>> processor uses the okHTTP library and ClientServicesImpl uses the Elastic
>>>> Search RestClient which in turn uses the HTTPClient from Apache.
>>>>
>>>> There was an issue with the HTTPClient library in v4.5.2 (
>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t
>>>> do the right hostname verification.
>>>>
>>>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use
>>>> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of
>>>> Apache HTTPClient.
>>>>
>>>>
>>>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties
>>>>
>>>> But again if it was a HostName validation issue the error message
>>>> should read more like: “Certificate for <host> doesn’t match common name of
>>>> certificate subject…”
>>>>
>>>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>>>> wrote:
>>>>
>>>> I'm far from a SSL/TLS expert, but let's get these out of the way:
>>>>
>>>> 1. Did you mark the server's cert as "trusted" when you created the
>>>> trust store with keytool?
>>>> 2. Are you sure that you're specifying the same hostname in the client
>>>> service that is in the CN field in the server's cert?
>>>>
>>>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can
>>>> use it to generate valid certificates. It's meant for NiFi, but the certs
>>>> it generates should be a solid step up from self-signed certs if you're not
>>>> able to access a company CA.
>>>>
>>>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com>
>>>> wrote:
>>>>
>>>>> Nope. No good. I even dump the network traffic and analyzed it in
>>>>> Wireshark. The ES server sends back two certificates (server + self-signed
>>>>> one) and both of them are present in my TrustStore. I am specifying both a
>>>>> TrustStore and a Keystore now but it still gives the error that it can’t
>>>>> find the certificate.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>,
>>>>> wrote:
>>>>>
>>>>> Think I might have found the issue. Will report tonight.
>>>>>
>>>>> Mike, please don’t spend any time debugging this because I think it
>>>>> might be an issue on my side. Appreciate all the help so far.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>,
>>>>> wrote:
>>>>>
>>>>> Here it is:
>>>>>
>>>>>
>>>>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7]
>>>>> o.a.n.processors.standard.LookupRecord
>>>>> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process
>>>>> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim
>>>>> [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default,
>>>>> section=1], offset=7180,
>>>>> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]:
>>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>>>>> coordinates {key=test} in Lookup Service
>>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>>>>> coordinates {key=test} in Lookup Service
>>>>> at
>>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
>>>>> at
>>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
>>>>> at
>>>>> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
>>>>> at
>>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
>>>>> at
>>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
>>>>> at
>>>>> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
>>>>> at
>>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>>>> at
>>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>>>>> at
>>>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
>>>>> at
>>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>>>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>>>>> org.apache.nifi.lookup.LookupFailureException:
>>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>> at
>>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
>>>>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at
>>>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>>>>> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
>>>>> at
>>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
>>>>> ... 17 common frames omitted
>>>>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>> at
>>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
>>>>> at
>>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
>>>>> ... 23 common frames omitted
>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>>> problem
>>>>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
>>>>> at
>>>>> sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
>>>>> at
>>>>> sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
>>>>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
>>>>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
>>>>> at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
>>>>> at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
>>>>> at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
>>>>> at
>>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
>>>>> ... 1 common frames omitted
>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>>> problem
>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>>>> at
>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>>> at
>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
>>>>> at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>>> at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>>> ... 9 common frames omitted
>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path
>>>>> building failed:
>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>>> valid certification path to requested target
>>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>>>>> at
>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>>> at sun.security.validator.Validator.validate(Validator.java:262)
>>>>> at
>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>>> at
>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>>> at
>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>>> at
>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>>> ... 17 common frames omitted
>>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>>>> unable to find valid certification path to requested target
>>>>> at
>>>>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>>>>> at
>>>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>>>>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
>>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
>>>>> ... 23 common frames omitted
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>,
>>>>> wrote:
>>>>>
>>>>> Can you share the stacktrace from the logs?
>>>>>
>>>>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Mike,
>>>>>>
>>>>>> The SSLContextService only had the Trust store configured. I think I
>>>>>> seen that ticket before but didn’t pay attention to the fact it wasn’t
>>>>>> merged in to the code I am running.
>>>>>>
>>>>>> However, I configured the service to have a KeyStore now but I am
>>>>>> getting the same errors…
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Peter
>>>>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>>>>>> wrote:
>>>>>>
>>>>>> Peter,
>>>>>>
>>>>>> Are you configuring the service as a trust-only configuration? If so,
>>>>>> that's been addressed in the 1.10 which is due for release in the near(ish)
>>>>>> future.
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/NIFI-6228
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> As a follow-up.
>>>>>>>
>>>>>>> On the Nifi node I am able to do a GET to Elastic Search using curl.
>>>>>>> I specify the —cacert option giving it the self-signed root certificate.
>>>>>>>
>>>>>>> Of course, this isn’t using the TrustStore but I am able to use the
>>>>>>> TrustStore if I use other ES processors… just not the
>>>>>>> ElasticSearchClientServicesImpl.
>>>>>>>
>>>>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <
>>>>>>> peter.moberg@gmail.com>, wrote:
>>>>>>>
>>>>>>> Hi Andy,
>>>>>>>
>>>>>>> thanks for your suggestions. Here is what I have tried so far (still
>>>>>>> no luck).
>>>>>>>
>>>>>>> Connecting with openssl and viewing the certs it presents
>>>>>>>
>>>>>>> *openssl s_client -connect quickstart-es-http.es
>>>>>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>>>>>
>>>>>>> If I then look inside the server cert I can find this
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>>>>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>>>>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>>>>>
>>>>>>>
>>>>>>> If I look in to the self-signed root cert I find this:
>>>>>>>
>>>>>>>
>>>>>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>>>>>
>>>>>>>
>>>>>>> I now double check  my trust store to make sure the Root Cert is
>>>>>>> there.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Trust store content Your keystore contains 1 entry Alias name:
>>>>>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>>>>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>>>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>>>>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>>>>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>>>>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>>>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>>>>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>>>>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>>>>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>>>>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>>>>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>>>>>> DigitalSignature Key_CertSign ]*
>>>>>>>
>>>>>>> So everything looks Ok. But when I run the
>>>>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>>>>>> store I still get the following output in the log.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>>>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>>>>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>>>>> at
>>>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>>>>> at
>>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>>>>> ... 9 common frames omitted Caused by:
>>>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>>>>> valid certification path to requested target at
>>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>>>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>>>>> at
>>>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>>>>> at
>>>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>>>>> at
>>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>>>>> ... 17 common frames omitted*
>>>>>>>
>>>>>>> Both the Nifi install and Elastic Search install is running in
>>>>>>> Kubernetes. The address I am using is a service address that is backed by 3
>>>>>>> ES instances. However, I double checked all three of the ES nodes to make
>>>>>>> sure that they returned back the same SSL cert and they did.
>>>>>>>
>>>>>>> The only thing I haven't been able to figure out is how to check if
>>>>>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>>>>>> that it would return different SSL certs but stranger things have happened…
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Peter,
>>>>>>>
>>>>>>> If you can use openssl’s s_client command (example below) to connect
>>>>>>> to the endpoint and verify that the hostname matches the certificate and
>>>>>>> that the certificate contains a SubjectAlternativeName entry with that
>>>>>>> hostname (see RFC 6125 [1] for more details), this should help you debug
>>>>>>> the issue. The cause of the PKIX error is that the truststore doesn’t
>>>>>>> contain a certificate (or certificate chain) which matches the hostname
>>>>>>> presented by the remote endpoint. I think you understand that based on your
>>>>>>> message. The underlying reason for this is could be one of the following:
>>>>>>>
>>>>>>> * the server is behind an interface which responds differently to
>>>>>>> GET and POST/PUT requests
>>>>>>> * there is a load-balancer which is directing the requests
>>>>>>> coincidentally to different backend servers (one has the right cert; the
>>>>>>> other doesn’t)
>>>>>>> * I recall something around the addition of (some) Elastic Search
>>>>>>> components which handled TLS in an ES client-specific manner; I remember
>>>>>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>>>>>> was ultimately contributed. If it’s not one of the above issues, I can
>>>>>>> investigate further.
>>>>>>>
>>>>>>> Hopefully this helps.
>>>>>>>
>>>>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>>>>>
>>>>>>> s_client example:
>>>>>>>
>>>>>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>>>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>>>>>> <path_to_your_CA_cert.pem>
>>>>>>>
>>>>>>> Andy LoPresto
>>>>>>> alopresto@apache.org
>>>>>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>>>>
>>>>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>>>>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>>>>>> flow that has the PutElasticSearchHttp component. I have setup a
>>>>>>> SSLContextService for that component where I have specified a trust store
>>>>>>> that has the self-signed cert from ES. I specify an https endpoint to
>>>>>>> access Elastic Search and Im having no issues populating my Elastic Search
>>>>>>> instance using this flow.
>>>>>>>
>>>>>>> I have another flow where I want to do some lookups. So I have been
>>>>>>> using the LookupRecord processor. That one I have associated with an
>>>>>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>>>>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>>>>>> checked this). However, when I run this second Flow I am not able to verify
>>>>>>> the ES server's self-signed certificate.
>>>>>>>
>>>>>>> I check the nifi-app.log and it says:
>>>>>>> Caused by:
>>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>>>>> valid certification path to requested target
>>>>>>>
>>>>>>> I am a bit surprised that I am not able to verify the same server
>>>>>>> certificate in the two different flows.
>>>>>>>
>>>>>>> Completely stuck on this so if anyone have any pointers please let
>>>>>>> me know.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Peter
>>>>>>>
>>>>>>>
>>>>>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Matt Burgess <ma...@gmail.com>.
For saving space (if you want to run from the assembly/target directory) you can add a -Ddir-only and it won’t build the tar.gz.




Sent from my iPhone
> On Oct 22, 2019, at 2:25 PM, Mike Thomsen <mi...@gmail.com> wrote:
> 
> 
> If you run `mvn clean install -DskipTests=true` from the root of the source folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd recommend testing against that as it'll be faster.
> 
>> On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg <pe...@gmail.com> wrote:
>> Mike,
>> 
>> thanks for putting together that PR. I have built everything successfully but I haven't been able to test this yet since I haven't built the new Docker image.
>> 
>> I assume you guys just use the ‘dockermaven’  folder to build the Dockerfile with src artifacts? 
>> 
>> My current dev machine is running out of disk space so I am spending some time trying to get it back to a runnable state. Hopefully I can build the PR and test it in Kubernetes this week sometime.
>> 
>> Thanks,
>> 
>> Peter
>>> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
>>> As a compromise, I upgraded to the latest 5.X client and manually incremented Apache HttpClient to 4.5.10. PR is here:
>>> 
>>> https://github.com/apache/nifi/pull/3828
>>> 
>>> There are integration tests for that package that automatically startup and provision an ES node, and they all passed with this configuration. Hopefully this PR will fix your issue since it appears to be an external dependency causing it.
>>> 
>>>> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mi...@gmail.com> wrote:
>>>> There's no hard and fast reason to stay with 5.X there, so you can build your own copy of 1.9.2 with that dependency upgraded if you want to try it out. I'll try to find time to test that change on 1.10.0-SNAPSHOT.
>>>> 
>>>>> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <pe...@gmail.com> wrote:
>>>>> The certs in the TrustStore are marked as trusted. 
>>>>> 
>>>>> The host name specified in the ClientServiceImpl is: 
>>>>> https://quickstart-es-http.es-cluster:9200
>>>>> 
>>>>> The CN field of the server certificate is:
>>>>> https://quickstart-es-http.es-cluster.es.local
>>>>> 
>>>>> So at first glance it looks like the issue would be that the CN field differs from the host name. However, Im not getting an error message saying that the HostName doesn’t match the CN. According to the RFC spec first mentioned by Andy the CN field should only be used if the SubjectiveAlternativeName is empty. 
>>>>> 
>>>>> The Server Cert has the SAN set to:  quickstart-es-http.es-cluster
>>>>> 
>>>>> What is really strange to me is why I can connect using the QueryElasticSearch processor with the same SSLContextService and hostname but when I use the ClientServicesImpl it will not work. 
>>>>> 
>>>>> So I did some more investigation and it looks like the QueryES processor uses the okHTTP library and ClientServicesImpl uses the Elastic Search RestClient which in turn uses the HTTPClient from Apache. 
>>>>> 
>>>>> There was an issue with the HTTPClient library in v4.5.2 (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t do the right hostname verification. 
>>>>> 
>>>>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of Apache HTTPClient.
>>>>> 
>>>>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties
>>>>> 
>>>>> But again if it was a HostName validation issue the error message should read more like: “Certificate for <host> doesn’t match common name of certificate subject…”
>>>>> 
>>>>>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
>>>>>> I'm far from a SSL/TLS expert, but let's get these out of the way:
>>>>>> 
>>>>>> 1. Did you mark the server's cert as "trusted" when you created the trust store with keytool?
>>>>>> 2. Are you sure that you're specifying the same hostname in the client service that is in the CN field in the server's cert?
>>>>>> 
>>>>>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use it to generate valid certificates. It's meant for NiFi, but the certs it generates should be a solid step up from self-signed certs if you're not able to access a company CA.
>>>>>> 
>>>>>>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com> wrote:
>>>>>>> Nope. No good. I even dump the network traffic and analyzed it in Wireshark. The ES server sends back two certificates (server + self-signed one) and both of them are present in my TrustStore. I am specifying both a TrustStore and a Keystore now but it still gives the error that it can’t find the certificate. 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Peter
>>>>>>>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
>>>>>>>> Think I might have found the issue. Will report tonight. 
>>>>>>>> 
>>>>>>>> Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Peter
>>>>>>>>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
>>>>>>>>> Here it is:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default, section=1], offset=7180, length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
>>>>>>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
>>>>>>>>> at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
>>>>>>>>> at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
>>>>>>>>> at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
>>>>>>>>> at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
>>>>>>>>> at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
>>>>>>>>> at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
>>>>>>>>> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>>>>>>>> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>>>>>>>>> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
>>>>>>>>> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>>>>>>>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>> Caused by: org.apache.nifi.lookup.LookupFailureException: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>>>>>> at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
>>>>>>>>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
>>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>>> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>>>>>>>>> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
>>>>>>>>> at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
>>>>>>>>> ... 17 common frames omitted
>>>>>>>>> Caused by: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>>>>>> at org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
>>>>>>>>> at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
>>>>>>>>> ... 23 common frames omitted
>>>>>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>>>>>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
>>>>>>>>> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
>>>>>>>>> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
>>>>>>>>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
>>>>>>>>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
>>>>>>>>> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
>>>>>>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
>>>>>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
>>>>>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
>>>>>>>>> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
>>>>>>>>> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
>>>>>>>>> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
>>>>>>>>> ... 1 common frames omitted
>>>>>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>>>>>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
>>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>>>>>>>> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>>>>>>> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
>>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>>>>>>> ... 9 common frames omitted
>>>>>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
>>>>>>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>>>>>>>>> at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>>>>>>> at sun.security.validator.Validator.validate(Validator.java:262)
>>>>>>>>> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>>>>>>> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>>>>>>> ... 17 common frames omitted
>>>>>>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
>>>>>>>>> at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>>>>>>>>> at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>>>>>>>>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
>>>>>>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
>>>>>>>>> ... 23 common frames omitted
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Peter
>>>>>>>>>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
>>>>>>>>>> Can you share the stacktrace from the logs?
>>>>>>>>>> 
>>>>>>>>>>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:
>>>>>>>>>>> Mike,
>>>>>>>>>>> 
>>>>>>>>>>> The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running. 
>>>>>>>>>>> 
>>>>>>>>>>> However, I configured the service to have a KeyStore now but I am getting the same errors… 
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Peter
>>>>>>>>>>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
>>>>>>>>>>>> Peter,
>>>>>>>>>>>> 
>>>>>>>>>>>> Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
>>>>>>>>>>>> 
>>>>>>>>>>>> https://issues.apache.org/jira/browse/NIFI-6228
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Mike
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
>>>>>>>>>>>>> As a follow-up.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate. 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl. 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
>>>>>>>>>>>>>> Hi Andy, 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> thanks for your suggestions. Here is what I have tried so far (still no luck). 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Connecting with openssl and viewing the certs it presents 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> openssl s_client -connect quickstart-es-http.es-cluster -showcerts
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If I then look inside the server cert I can find this
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Server Cert:
>>>>>>>>>>>>>> Issuer: OU = quickstart, CN = quickstart-http
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> X509v3 Subject Alternative Name:
>>>>>>>>>>>>>> DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If I look in to the self-signed root cert I find this:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Root Cert:
>>>>>>>>>>>>>> Subject: OU = quickstart, CN = quickstart-http
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I now double check  my trust store to make sure the Root Cert is there.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Trust store content
>>>>>>>>>>>>>> Your keystore contains 1 entry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Alias name: ca_elastic
>>>>>>>>>>>>>> Creation date: Oct 16, 2019
>>>>>>>>>>>>>> Entry type: trustedCertEntry
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Owner: CN=quickstart-http, OU=quickstart
>>>>>>>>>>>>>> Issuer: CN=quickstart-http, OU=quickstart
>>>>>>>>>>>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
>>>>>>>>>>>>>> Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
>>>>>>>>>>>>>> Certificate fingerprints:
>>>>>>>>>>>>>> MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
>>>>>>>>>>>>>> SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
>>>>>>>>>>>>>> SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>>>>>>>>>>>>> Signature algorithm name: SHA256withRSA
>>>>>>>>>>>>>> Subject Public Key Algorithm: 2048-bit RSA key
>>>>>>>>>>>>>> Version: 3
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Extensions:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> #1: ObjectId: 2.5.29.19 Criticality=true
>>>>>>>>>>>>>> BasicConstraints:[
>>>>>>>>>>>>>> CA:true
>>>>>>>>>>>>>> PathLen:2147483647
>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> #2: ObjectId: 2.5.29.37 Criticality=false
>>>>>>>>>>>>>> ExtendedKeyUsages [
>>>>>>>>>>>>>> serverAuth
>>>>>>>>>>>>>> clientAuth
>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> #3: ObjectId: 2.5.29.15 Criticality=true
>>>>>>>>>>>>>> KeyUsage [
>>>>>>>>>>>>>> DigitalSignature
>>>>>>>>>>>>>> Key_CertSign
>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log. 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>>>>>>>>>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>>>>>>>>>>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>>>>>>>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
>>>>>>>>>>>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>>>>>>>>>>>>> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>>>>>>>>>>>> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>>>>>>>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>>>>>>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
>>>>>>>>>>>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
>>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>>>>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
>>>>>>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>>>>>>>>>>>> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>>>>>>>>>>>> ... 9 common frames omitted
>>>>>>>>>>>>>> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
>>>>>>>>>>>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>>>>>>>>>>>>>> at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>>>>>>>>>>>> at sun.security.validator.Validator.validate(Validator.java:262)
>>>>>>>>>>>>>> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>>>>>>>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>>>>>>>>>>>> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>>>>>>>>>>>> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>>>>>>>>>>>> ... 17 common frames omitted
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did. 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened… 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
>>>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> * the server is behind an interface which responds differently to GET and POST/PUT requests
>>>>>>>>>>>>>>> * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
>>>>>>>>>>>>>>> * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further. 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hopefully this helps. 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> s_client example: 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Andy LoPresto
>>>>>>>>>>>>>>> alopresto@apache.org
>>>>>>>>>>>>>>> alopresto.apache@gmail.com
>>>>>>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I check the nifi-app.log and it says:
>>>>>>>>>>>>>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Completely stuck on this so if anyone have any pointers please let me know.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
If you run `mvn clean install -DskipTests=true` from the root of the source
folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd
recommend testing against that as it'll be faster.

On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg <pe...@gmail.com> wrote:

> Mike,
>
> thanks for putting together that PR. I have built everything successfully
> but I haven't been able to test this yet since I haven't built the new
> Docker image.
>
> I assume you guys just use the ‘dockermaven’  folder to build the
> Dockerfile with src artifacts?
>
> My current dev machine is running out of disk space so I am spending some
> time trying to get it back to a runnable state. Hopefully I can build the
> PR and test it in Kubernetes this week sometime.
>
> Thanks,
>
> Peter
> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mi...@gmail.com>,
> wrote:
>
> As a compromise, I upgraded to the latest 5.X client and manually
> incremented Apache HttpClient to 4.5.10. PR is here:
>
> https://github.com/apache/nifi/pull/3828
>
> There are integration tests for that package that automatically startup
> and provision an ES node, and they all passed with this configuration.
> Hopefully this PR will fix your issue since it appears to be an external
> dependency causing it.
>
> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> There's no hard and fast reason to stay with 5.X there, so you can build
>> your own copy of 1.9.2 with that dependency upgraded if you want to try it
>> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT.
>>
>> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <pe...@gmail.com>
>> wrote:
>>
>>> The certs in the TrustStore are marked as trusted.
>>>
>>> The host name specified in the ClientServiceImpl is:
>>> https://quickstart-es-http.es-cluster:9200
>>>
>>> The CN field of the server certificate is:
>>> https://quickstart-es-http.es-cluster.es.local
>>>
>>> So at first glance it looks like the issue would be that the CN field
>>> differs from the host name. However, Im not getting an error message saying
>>> that the HostName doesn’t match the CN. According to the RFC spec first
>>> mentioned by Andy the CN field should only be used if the
>>> SubjectiveAlternativeName is empty.
>>>
>>> The Server Cert has the SAN set to:  quickstart-es-http.es-cluster
>>>
>>> What is really strange to me is why I can connect using the
>>> QueryElasticSearch processor with the same SSLContextService and hostname
>>> but when I use the ClientServicesImpl it will not work.
>>>
>>> So I did some more investigation and it looks like the QueryES processor
>>> uses the okHTTP library and ClientServicesImpl uses the Elastic Search
>>> RestClient which in turn uses the HTTPClient from Apache.
>>>
>>> There was an issue with the HTTPClient library in v4.5.2 (
>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t
>>> do the right hostname verification.
>>>
>>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use
>>> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of
>>> Apache HTTPClient.
>>>
>>>
>>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties
>>>
>>> But again if it was a HostName validation issue the error message should
>>> read more like: “Certificate for <host> doesn’t match common name of
>>> certificate subject…”
>>>
>>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>>> wrote:
>>>
>>> I'm far from a SSL/TLS expert, but let's get these out of the way:
>>>
>>> 1. Did you mark the server's cert as "trusted" when you created the
>>> trust store with keytool?
>>> 2. Are you sure that you're specifying the same hostname in the client
>>> service that is in the CN field in the server's cert?
>>>
>>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can
>>> use it to generate valid certificates. It's meant for NiFi, but the certs
>>> it generates should be a solid step up from self-signed certs if you're not
>>> able to access a company CA.
>>>
>>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>>> Nope. No good. I even dump the network traffic and analyzed it in
>>>> Wireshark. The ES server sends back two certificates (server + self-signed
>>>> one) and both of them are present in my TrustStore. I am specifying both a
>>>> TrustStore and a Keystore now but it still gives the error that it can’t
>>>> find the certificate.
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>,
>>>> wrote:
>>>>
>>>> Think I might have found the issue. Will report tonight.
>>>>
>>>> Mike, please don’t spend any time debugging this because I think it
>>>> might be an issue on my side. Appreciate all the help so far.
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>,
>>>> wrote:
>>>>
>>>> Here it is:
>>>>
>>>>
>>>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7]
>>>> o.a.n.processors.standard.LookupRecord
>>>> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process
>>>> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim
>>>> [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default,
>>>> section=1], offset=7180,
>>>> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]:
>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>>>> coordinates {key=test} in Lookup Service
>>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>>>> coordinates {key=test} in Lookup Service
>>>> at
>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
>>>> at
>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
>>>> at
>>>> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
>>>> at
>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
>>>> at
>>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
>>>> at
>>>> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
>>>> at
>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>>> at
>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>>>> at
>>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
>>>> at
>>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>>>> org.apache.nifi.lookup.LookupFailureException:
>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>> at
>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
>>>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>>>> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
>>>> at
>>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
>>>> ... 17 common frames omitted
>>>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>>> at
>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
>>>> at
>>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
>>>> ... 23 common frames omitted
>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>> problem
>>>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
>>>> at
>>>> sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
>>>> at
>>>> sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
>>>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
>>>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
>>>> at
>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
>>>> at
>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
>>>> ... 1 common frames omitted
>>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>> problem
>>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
>>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
>>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>> ... 9 common frames omitted
>>>> Caused by: sun.security.validator.ValidatorException: PKIX path
>>>> building failed:
>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>> valid certification path to requested target
>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>>>> at
>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>> at sun.security.validator.Validator.validate(Validator.java:262)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>> ... 17 common frames omitted
>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>>> unable to find valid certification path to requested target
>>>> at
>>>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>>>> at
>>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>>>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
>>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
>>>> ... 23 common frames omitted
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>,
>>>> wrote:
>>>>
>>>> Can you share the stacktrace from the logs?
>>>>
>>>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com>
>>>> wrote:
>>>>
>>>>> Mike,
>>>>>
>>>>> The SSLContextService only had the Trust store configured. I think I
>>>>> seen that ticket before but didn’t pay attention to the fact it wasn’t
>>>>> merged in to the code I am running.
>>>>>
>>>>> However, I configured the service to have a KeyStore now but I am
>>>>> getting the same errors…
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>>>>> wrote:
>>>>>
>>>>> Peter,
>>>>>
>>>>> Are you configuring the service as a trust-only configuration? If so,
>>>>> that's been addressed in the 1.10 which is due for release in the near(ish)
>>>>> future.
>>>>>
>>>>> https://issues.apache.org/jira/browse/NIFI-6228
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike
>>>>>
>>>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> As a follow-up.
>>>>>>
>>>>>> On the Nifi node I am able to do a GET to Elastic Search using curl.
>>>>>> I specify the —cacert option giving it the self-signed root certificate.
>>>>>>
>>>>>> Of course, this isn’t using the TrustStore but I am able to use the
>>>>>> TrustStore if I use other ES processors… just not the
>>>>>> ElasticSearchClientServicesImpl.
>>>>>>
>>>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>>>>>> wrote:
>>>>>>
>>>>>> Hi Andy,
>>>>>>
>>>>>> thanks for your suggestions. Here is what I have tried so far (still
>>>>>> no luck).
>>>>>>
>>>>>> Connecting with openssl and viewing the certs it presents
>>>>>>
>>>>>> *openssl s_client -connect quickstart-es-http.es
>>>>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>>>>
>>>>>> If I then look inside the server cert I can find this
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>>>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>>>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>>>>
>>>>>>
>>>>>> If I look in to the self-signed root cert I find this:
>>>>>>
>>>>>>
>>>>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>>>>
>>>>>>
>>>>>> I now double check  my trust store to make sure the Root Cert is
>>>>>> there.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Trust store content Your keystore contains 1 entry Alias name:
>>>>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>>>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>>>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>>>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>>>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>>>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>>>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>>>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>>>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>>>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>>>>> DigitalSignature Key_CertSign ]*
>>>>>>
>>>>>> So everything looks Ok. But when I run the
>>>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>>>>> store I still get the following output in the log.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>>>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>>>> at
>>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>>>> at
>>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>>>> ... 9 common frames omitted Caused by:
>>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>>>> valid certification path to requested target at
>>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>>>> at
>>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>>>> at
>>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>>>> at
>>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>>>> ... 17 common frames omitted*
>>>>>>
>>>>>> Both the Nifi install and Elastic Search install is running in
>>>>>> Kubernetes. The address I am using is a service address that is backed by 3
>>>>>> ES instances. However, I double checked all three of the ES nodes to make
>>>>>> sure that they returned back the same SSL cert and they did.
>>>>>>
>>>>>> The only thing I haven't been able to figure out is how to check if
>>>>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>>>>> that it would return different SSL certs but stranger things have happened…
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>>>>> wrote:
>>>>>>
>>>>>> Hi Peter,
>>>>>>
>>>>>> If you can use openssl’s s_client command (example below) to connect
>>>>>> to the endpoint and verify that the hostname matches the certificate and
>>>>>> that the certificate contains a SubjectAlternativeName entry with that
>>>>>> hostname (see RFC 6125 [1] for more details), this should help you debug
>>>>>> the issue. The cause of the PKIX error is that the truststore doesn’t
>>>>>> contain a certificate (or certificate chain) which matches the hostname
>>>>>> presented by the remote endpoint. I think you understand that based on your
>>>>>> message. The underlying reason for this is could be one of the following:
>>>>>>
>>>>>> * the server is behind an interface which responds differently to GET
>>>>>> and POST/PUT requests
>>>>>> * there is a load-balancer which is directing the requests
>>>>>> coincidentally to different backend servers (one has the right cert; the
>>>>>> other doesn’t)
>>>>>> * I recall something around the addition of (some) Elastic Search
>>>>>> components which handled TLS in an ES client-specific manner; I remember
>>>>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>>>>> was ultimately contributed. If it’s not one of the above issues, I can
>>>>>> investigate further.
>>>>>>
>>>>>> Hopefully this helps.
>>>>>>
>>>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>>>>
>>>>>> s_client example:
>>>>>>
>>>>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>>>>> <path_to_your_CA_cert.pem>
>>>>>>
>>>>>> Andy LoPresto
>>>>>> alopresto@apache.org
>>>>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>>>
>>>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>>>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>>>>> flow that has the PutElasticSearchHttp component. I have setup a
>>>>>> SSLContextService for that component where I have specified a trust store
>>>>>> that has the self-signed cert from ES. I specify an https endpoint to
>>>>>> access Elastic Search and Im having no issues populating my Elastic Search
>>>>>> instance using this flow.
>>>>>>
>>>>>> I have another flow where I want to do some lookups. So I have been
>>>>>> using the LookupRecord processor. That one I have associated with an
>>>>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>>>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>>>>> checked this). However, when I run this second Flow I am not able to verify
>>>>>> the ES server's self-signed certificate.
>>>>>>
>>>>>> I check the nifi-app.log and it says:
>>>>>> Caused by:
>>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>>>> valid certification path to requested target
>>>>>>
>>>>>> I am a bit surprised that I am not able to verify the same server
>>>>>> certificate in the two different flows.
>>>>>>
>>>>>> Completely stuck on this so if anyone have any pointers please let me
>>>>>> know.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>>
>>>>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
Mike,

thanks for putting together that PR. I have built everything successfully but I haven't been able to test this yet since I haven't built the new Docker image.

I assume you guys just use the ‘dockermaven’  folder to build the Dockerfile with src artifacts?

My current dev machine is running out of disk space so I am spending some time trying to get it back to a runnable state. Hopefully I can build the PR and test it in Kubernetes this week sometime.

Thanks,

Peter
On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> As a compromise, I upgraded to the latest 5.X client and manually incremented Apache HttpClient to 4.5.10. PR is here:
>
> https://github.com/apache/nifi/pull/3828
>
> There are integration tests for that package that automatically startup and provision an ES node, and they all passed with this configuration. Hopefully this PR will fix your issue since it appears to be an external dependency causing it.
>
> > On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mi...@gmail.com> wrote:
> > > There's no hard and fast reason to stay with 5.X there, so you can build your own copy of 1.9.2 with that dependency upgraded if you want to try it out. I'll try to find time to test that change on 1.10.0-SNAPSHOT.
> > >
> > > > On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > The certs in the TrustStore are marked as trusted.
> > > > >
> > > > > The host name specified in the ClientServiceImpl is:
> > > > > https://quickstart-es-http.es-cluster:9200
> > > > >
> > > > > The CN field of the server certificate is:
> > > > > https://quickstart-es-http.es-cluster.es.local
> > > > >
> > > > > So at first glance it looks like the issue would be that the CN field differs from the host name. However, Im not getting an error message saying that the HostName doesn’t match the CN. According to the RFC spec first mentioned by Andy the CN field should only be used if the SubjectiveAlternativeName is empty.
> > > > >
> > > > > The Server Cert has the SAN set to:  quickstart-es-http.es-cluster
> > > > >
> > > > > What is really strange to me is why I can connect using the QueryElasticSearch processor with the same SSLContextService and hostname but when I use the ClientServicesImpl it will not work.
> > > > >
> > > > > So I did some more investigation and it looks like the QueryES processor uses the okHTTP library and ClientServicesImpl uses the Elastic Search RestClient which in turn uses the HTTPClient from Apache.
> > > > >
> > > > > There was an issue with the HTTPClient library in v4.5.2 (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t do the right hostname verification.
> > > > >
> > > > > The version of Apache Nifi that Im using is 1.9.2 and it seems to use v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of Apache HTTPClient.
> > > > >
> > > > > https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties
> > > > >
> > > > > But again if it was a HostName validation issue the error message should read more like: “Certificate for <host> doesn’t match common name of certificate subject…”
> > > > >
> > > > > On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > > I'm far from a SSL/TLS expert, but let's get these out of the way:
> > > > > >
> > > > > > 1. Did you mark the server's cert as "trusted" when you created the trust store with keytool?
> > > > > > 2. Are you sure that you're specifying the same hostname in the client service that is in the CN field in the server's cert?
> > > > > >
> > > > > > FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use it to generate valid certificates. It's meant for NiFi, but the certs it generates should be a solid step up from self-signed certs if you're not able to access a company CA.
> > > > > >
> > > > > > > On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > Nope. No good. I even dump the network traffic and analyzed it in Wireshark. The ES server sends back two certificates (server + self-signed one) and both of them are present in my TrustStore. I am specifying both a TrustStore and a Keystore now but it still gives the error that it can’t find the certificate.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Peter
> > > > > > > > On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > > > Think I might have found the issue. Will report tonight.
> > > > > > > > >
> > > > > > > > > Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Peter
> > > > > > > > > On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > > > > Here it is:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default, section=1], offset=7180, length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> > > > > > > > > > org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> > > > > > > > > > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
> > > > > > > > > > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
> > > > > > > > > > at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
> > > > > > > > > > at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
> > > > > > > > > > at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
> > > > > > > > > > at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
> > > > > > > > > > at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> > > > > > > > > > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> > > > > > > > > > at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> > > > > > > > > > at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> > > > > > > > > > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> > > > > > > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > > > > > Caused by: org.apache.nifi.lookup.LookupFailureException: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
> > > > > > > > > > at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
> > > > > > > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > > > > > at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> > > > > > > > > > at com.sun.proxy.$Proxy136.lookup(Unknown Source)
> > > > > > > > > > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
> > > > > > > > > > ... 17 common frames omitted
> > > > > > > > > > Caused by: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
> > > > > > > > > > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
> > > > > > > > > > ... 23 common frames omitted
> > > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > > at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> > > > > > > > > > at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> > > > > > > > > > at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> > > > > > > > > > at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> > > > > > > > > > at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
> > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
> > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> > > > > > > > > > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> > > > > > > > > > ... 1 common frames omitted
> > > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > > > > ... 9 common frames omitted
> > > > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > > > > ... 17 common frames omitted
> > > > > > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> > > > > > > > > > at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> > > > > > > > > > at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> > > > > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> > > > > > > > > > ... 23 common frames omitted
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Peter
> > > > > > > > > > On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > > > > > > > Can you share the stacktrace from the logs?
> > > > > > > > > > >
> > > > > > > > > > > > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > > > > > Mike,
> > > > > > > > > > > > >
> > > > > > > > > > > > > The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running.
> > > > > > > > > > > > >
> > > > > > > > > > > > > However, I configured the service to have a KeyStore now but I am getting the same errors…
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Peter
> > > > > > > > > > > > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > > > > > > > > > > Peter,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/NIFI-6228
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > > > > > > > > As a follow-up.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > > > > > > > > > > > Hi Andy,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > thanks for your suggestions. Here is what I have tried so far (still no luck).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Connecting with openssl and viewing the certs it presents
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > openssl s_client -connect quickstart-es-http.es-cluster -showcerts
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If I then look inside the server cert I can find this
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Server Cert:
> > > > > > > > > > > > > > > > > Issuer: OU = quickstart, CN = quickstart-http
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > X509v3 Subject Alternative Name:
> > > > > > > > > > > > > > > > > DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If I look in to the self-signed root cert I find this:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Root Cert:
> > > > > > > > > > > > > > > > > Subject: OU = quickstart, CN = quickstart-http
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I now double check  my trust store to make sure the Root Cert is there.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Trust store content
> > > > > > > > > > > > > > > > > Your keystore contains 1 entry
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Alias name: ca_elastic
> > > > > > > > > > > > > > > > > Creation date: Oct 16, 2019
> > > > > > > > > > > > > > > > > Entry type: trustedCertEntry
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Owner: CN=quickstart-http, OU=quickstart
> > > > > > > > > > > > > > > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > > > > > > > > > > > > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > > > > > > > > > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> > > > > > > > > > > > > > > > > Certificate fingerprints:
> > > > > > > > > > > > > > > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > > > > > > > > > > > > > > SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > > > > > > > > > > > > > > SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > > > > > > > > > > > > > > Signature algorithm name: SHA256withRSA
> > > > > > > > > > > > > > > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > > > > > > > > > > > > > > Version: 3
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Extensions:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > > > > > > > > > > > > > > BasicConstraints:[
> > > > > > > > > > > > > > > > > CA:true
> > > > > > > > > > > > > > > > > PathLen:2147483647
> > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > > > > > > > > > > > > > > ExtendedKeyUsages [
> > > > > > > > > > > > > > > > > serverAuth
> > > > > > > > > > > > > > > > > clientAuth
> > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > > > > > > > > > > > > > > KeyUsage [
> > > > > > > > > > > > > > > > > DigitalSignature
> > > > > > > > > > > > > > > > > Key_CertSign
> > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > > > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > > > > > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > > > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > > > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > > > > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > > > > > > > > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > > > > > > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > > > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > > > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > > > > > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > > > > > > > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > > > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > > > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > > > > > > > > > > > ... 9 common frames omitted
> > > > > > > > > > > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > > > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > > > > > > > > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > > > > > > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > > > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > > > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > > > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > > > > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > > > > > > > > > > > ... 17 common frames omitted
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > > > > > > > > > > > > > > > > > Hi Peter,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > > > > > > > > > > > > > > > > > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > > > > > > > > > > > > > > > > > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hopefully this helps.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > s_client example:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Andy LoPresto
> > > > > > > > > > > > > > > > > > alopresto@apache.org
> > > > > > > > > > > > > > > > > > alopresto.apache@gmail.com
> > > > > > > > > > > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I check the nifi-app.log and it says:
> > > > > > > > > > > > > > > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Completely stuck on this so if anyone have any pointers please let me know.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Peter
> > > > > > > > > > > > > > > > > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
As a compromise, I upgraded to the latest 5.X client and manually
incremented Apache HttpClient to 4.5.10. PR is here:

https://github.com/apache/nifi/pull/3828

There are integration tests for that package that automatically startup and
provision an ES node, and they all passed with this configuration.
Hopefully this PR will fix your issue since it appears to be an external
dependency causing it.

On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen <mi...@gmail.com> wrote:

> There's no hard and fast reason to stay with 5.X there, so you can build
> your own copy of 1.9.2 with that dependency upgraded if you want to try it
> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT.
>
> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <pe...@gmail.com>
> wrote:
>
>> The certs in the TrustStore are marked as trusted.
>>
>> The host name specified in the ClientServiceImpl is:
>> https://quickstart-es-http.es-cluster:9200
>>
>> The CN field of the server certificate is:
>> https://quickstart-es-http.es-cluster.es.local
>>
>> So at first glance it looks like the issue would be that the CN field
>> differs from the host name. However, Im not getting an error message saying
>> that the HostName doesn’t match the CN. According to the RFC spec first
>> mentioned by Andy the CN field should only be used if the
>> SubjectiveAlternativeName is empty.
>>
>> The Server Cert has the SAN set to:  quickstart-es-http.es-cluster
>>
>> What is really strange to me is why I can connect using the
>> QueryElasticSearch processor with the same SSLContextService and hostname
>> but when I use the ClientServicesImpl it will not work.
>>
>> So I did some more investigation and it looks like the QueryES processor
>> uses the okHTTP library and ClientServicesImpl uses the Elastic Search
>> RestClient which in turn uses the HTTPClient from Apache.
>>
>> There was an issue with the HTTPClient library in v4.5.2 (
>> https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t
>> do the right hostname verification.
>>
>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use
>> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of
>> Apache HTTPClient.
>>
>>
>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties
>>
>> But again if it was a HostName validation issue the error message should
>> read more like: “Certificate for <host> doesn’t match common name of
>> certificate subject…”
>>
>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>> wrote:
>>
>> I'm far from a SSL/TLS expert, but let's get these out of the way:
>>
>> 1. Did you mark the server's cert as "trusted" when you created the trust
>> store with keytool?
>> 2. Are you sure that you're specifying the same hostname in the client
>> service that is in the CN field in the server's cert?
>>
>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use
>> it to generate valid certificates. It's meant for NiFi, but the certs it
>> generates should be a solid step up from self-signed certs if you're not
>> able to access a company CA.
>>
>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com>
>> wrote:
>>
>>> Nope. No good. I even dump the network traffic and analyzed it in
>>> Wireshark. The ES server sends back two certificates (server + self-signed
>>> one) and both of them are present in my TrustStore. I am specifying both a
>>> TrustStore and a Keystore now but it still gives the error that it can’t
>>> find the certificate.
>>>
>>> Thanks,
>>>
>>> Peter
>>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>,
>>> wrote:
>>>
>>> Think I might have found the issue. Will report tonight.
>>>
>>> Mike, please don’t spend any time debugging this because I think it
>>> might be an issue on my side. Appreciate all the help so far.
>>>
>>> Thanks,
>>>
>>> Peter
>>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>,
>>> wrote:
>>>
>>> Here it is:
>>>
>>>
>>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7]
>>> o.a.n.processors.standard.LookupRecord
>>> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process
>>> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim
>>> [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default,
>>> section=1], offset=7180,
>>> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]:
>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>>> coordinates {key=test} in Lookup Service
>>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>>> coordinates {key=test} in Lookup Service
>>> at
>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
>>> at
>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
>>> at
>>> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
>>> at
>>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
>>> at
>>> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
>>> at
>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> at
>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>>> at
>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
>>> at
>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>>> org.apache.nifi.lookup.LookupFailureException:
>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>> at
>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
>>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>>> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
>>> at
>>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
>>> ... 17 common frames omitted
>>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>> at
>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
>>> at
>>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
>>> ... 23 common frames omitted
>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
>>> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
>>> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
>>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
>>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
>>> at
>>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
>>> at
>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
>>> at
>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
>>> ... 1 common frames omitted
>>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>> at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>> at
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
>>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>> ... 9 common frames omitted
>>> Caused by: sun.security.validator.ValidatorException: PKIX path building
>>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
>>> to find valid certification path to requested target
>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>>> at
>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>> at sun.security.validator.Validator.validate(Validator.java:262)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>> at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>> ... 17 common frames omitted
>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>> unable to find valid certification path to requested target
>>> at
>>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>>> at
>>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
>>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
>>> ... 23 common frames omitted
>>>
>>>
>>> Thanks,
>>>
>>> Peter
>>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>,
>>> wrote:
>>>
>>> Can you share the stacktrace from the logs?
>>>
>>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>>> Mike,
>>>>
>>>> The SSLContextService only had the Trust store configured. I think I
>>>> seen that ticket before but didn’t pay attention to the fact it wasn’t
>>>> merged in to the code I am running.
>>>>
>>>> However, I configured the service to have a KeyStore now but I am
>>>> getting the same errors…
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>>>> wrote:
>>>>
>>>> Peter,
>>>>
>>>> Are you configuring the service as a trust-only configuration? If so,
>>>> that's been addressed in the 1.10 which is due for release in the near(ish)
>>>> future.
>>>>
>>>> https://issues.apache.org/jira/browse/NIFI-6228
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>>
>>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>>>> wrote:
>>>>
>>>>> As a follow-up.
>>>>>
>>>>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>>>>> specify the —cacert option giving it the self-signed root certificate.
>>>>>
>>>>> Of course, this isn’t using the TrustStore but I am able to use the
>>>>> TrustStore if I use other ES processors… just not the
>>>>> ElasticSearchClientServicesImpl.
>>>>>
>>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>>>>> wrote:
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> thanks for your suggestions. Here is what I have tried so far (still
>>>>> no luck).
>>>>>
>>>>> Connecting with openssl and viewing the certs it presents
>>>>>
>>>>> *openssl s_client -connect quickstart-es-http.es
>>>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>>>
>>>>> If I then look inside the server cert I can find this
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>>>
>>>>>
>>>>> If I look in to the self-signed root cert I find this:
>>>>>
>>>>>
>>>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>>>
>>>>>
>>>>> I now double check  my trust store to make sure the Root Cert is there.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Trust store content Your keystore contains 1 entry Alias name:
>>>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>>>> DigitalSignature Key_CertSign ]*
>>>>>
>>>>> So everything looks Ok. But when I run the
>>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>>>> store I still get the following output in the log.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>>> at
>>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>>> at
>>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>>> ... 9 common frames omitted Caused by:
>>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>>> valid certification path to requested target at
>>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>>> at
>>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>>> at
>>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>>> at
>>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>>> ... 17 common frames omitted*
>>>>>
>>>>> Both the Nifi install and Elastic Search install is running in
>>>>> Kubernetes. The address I am using is a service address that is backed by 3
>>>>> ES instances. However, I double checked all three of the ES nodes to make
>>>>> sure that they returned back the same SSL cert and they did.
>>>>>
>>>>> The only thing I haven't been able to figure out is how to check if
>>>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>>>> that it would return different SSL certs but stranger things have happened…
>>>>>
>>>>>
>>>>>
>>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>>>> wrote:
>>>>>
>>>>> Hi Peter,
>>>>>
>>>>> If you can use openssl’s s_client command (example below) to connect
>>>>> to the endpoint and verify that the hostname matches the certificate and
>>>>> that the certificate contains a SubjectAlternativeName entry with that
>>>>> hostname (see RFC 6125 [1] for more details), this should help you debug
>>>>> the issue. The cause of the PKIX error is that the truststore doesn’t
>>>>> contain a certificate (or certificate chain) which matches the hostname
>>>>> presented by the remote endpoint. I think you understand that based on your
>>>>> message. The underlying reason for this is could be one of the following:
>>>>>
>>>>> * the server is behind an interface which responds differently to GET
>>>>> and POST/PUT requests
>>>>> * there is a load-balancer which is directing the requests
>>>>> coincidentally to different backend servers (one has the right cert; the
>>>>> other doesn’t)
>>>>> * I recall something around the addition of (some) Elastic Search
>>>>> components which handled TLS in an ES client-specific manner; I remember
>>>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>>>> was ultimately contributed. If it’s not one of the above issues, I can
>>>>> investigate further.
>>>>>
>>>>> Hopefully this helps.
>>>>>
>>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>>>
>>>>> s_client example:
>>>>>
>>>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>>>> <path_to_your_CA_cert.pem>
>>>>>
>>>>> Andy LoPresto
>>>>> alopresto@apache.org
>>>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>>
>>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>>>> flow that has the PutElasticSearchHttp component. I have setup a
>>>>> SSLContextService for that component where I have specified a trust store
>>>>> that has the self-signed cert from ES. I specify an https endpoint to
>>>>> access Elastic Search and Im having no issues populating my Elastic Search
>>>>> instance using this flow.
>>>>>
>>>>> I have another flow where I want to do some lookups. So I have been
>>>>> using the LookupRecord processor. That one I have associated with an
>>>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>>>> checked this). However, when I run this second Flow I am not able to verify
>>>>> the ES server's self-signed certificate.
>>>>>
>>>>> I check the nifi-app.log and it says:
>>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>>>> unable to find valid certification path to requested target
>>>>>
>>>>> I am a bit surprised that I am not able to verify the same server
>>>>> certificate in the two different flows.
>>>>>
>>>>> Completely stuck on this so if anyone have any pointers please let me
>>>>> know.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Peter
>>>>>
>>>>>
>>>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
There's no hard and fast reason to stay with 5.X there, so you can build
your own copy of 1.9.2 with that dependency upgraded if you want to try it
out. I'll try to find time to test that change on 1.10.0-SNAPSHOT.

On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg <pe...@gmail.com> wrote:

> The certs in the TrustStore are marked as trusted.
>
> The host name specified in the ClientServiceImpl is:
> https://quickstart-es-http.es-cluster:9200
>
> The CN field of the server certificate is:
> https://quickstart-es-http.es-cluster.es.local
>
> So at first glance it looks like the issue would be that the CN field
> differs from the host name. However, Im not getting an error message saying
> that the HostName doesn’t match the CN. According to the RFC spec first
> mentioned by Andy the CN field should only be used if the
> SubjectiveAlternativeName is empty.
>
> The Server Cert has the SAN set to:  quickstart-es-http.es-cluster
>
> What is really strange to me is why I can connect using the
> QueryElasticSearch processor with the same SSLContextService and hostname
> but when I use the ClientServicesImpl it will not work.
>
> So I did some more investigation and it looks like the QueryES processor
> uses the okHTTP library and ClientServicesImpl uses the Elastic Search
> RestClient which in turn uses the HTTPClient from Apache.
>
> There was an issue with the HTTPClient library in v4.5.2 (
> https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t do
> the right hostname verification.
>
> The version of Apache Nifi that Im using is 1.9.2 and it seems to use
> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of
> Apache HTTPClient.
>
>
> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties
>
> But again if it was a HostName validation issue the error message should
> read more like: “Certificate for <host> doesn’t match common name of
> certificate subject…”
>
> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
> wrote:
>
> I'm far from a SSL/TLS expert, but let's get these out of the way:
>
> 1. Did you mark the server's cert as "trusted" when you created the trust
> store with keytool?
> 2. Are you sure that you're specifying the same hostname in the client
> service that is in the CN field in the server's cert?
>
> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use
> it to generate valid certificates. It's meant for NiFi, but the certs it
> generates should be a solid step up from self-signed certs if you're not
> able to access a company CA.
>
> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com>
> wrote:
>
>> Nope. No good. I even dump the network traffic and analyzed it in
>> Wireshark. The ES server sends back two certificates (server + self-signed
>> one) and both of them are present in my TrustStore. I am specifying both a
>> TrustStore and a Keystore now but it still gives the error that it can’t
>> find the certificate.
>>
>> Thanks,
>>
>> Peter
>> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>,
>> wrote:
>>
>> Think I might have found the issue. Will report tonight.
>>
>> Mike, please don’t spend any time debugging this because I think it might
>> be an issue on my side. Appreciate all the help so far.
>>
>> Thanks,
>>
>> Peter
>> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>,
>> wrote:
>>
>> Here it is:
>>
>>
>> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7]
>> o.a.n.processors.standard.LookupRecord
>> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process
>> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default,
>> section=1], offset=7180,
>> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]:
>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>> coordinates {key=test} in Lookup Service
>> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
>> coordinates {key=test} in Lookup Service
>> at
>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
>> at
>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
>> at
>> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
>> at
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
>> at
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
>> at
>> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>> org.apache.nifi.lookup.LookupFailureException:
>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>> at
>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
>> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
>> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
>> at
>> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
>> ... 17 common frames omitted
>> Caused by: org.apache.nifi.lookup.LookupFailureException:
>> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>> at
>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
>> at
>> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
>> ... 23 common frames omitted
>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
>> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
>> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
>> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
>> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
>> at
>> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
>> at
>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
>> at
>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
>> ... 1 common frames omitted
>> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>> at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>> at
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
>> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>> ... 9 common frames omitted
>> Caused by: sun.security.validator.ValidatorException: PKIX path building
>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
>> to find valid certification path to requested target
>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
>> at
>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>> at sun.security.validator.Validator.validate(Validator.java:262)
>> at
>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>> at
>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>> at
>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>> at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>> ... 17 common frames omitted
>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable to find valid certification path to requested target
>> at
>> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
>> at
>> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
>> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
>> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
>> ... 23 common frames omitted
>>
>>
>> Thanks,
>>
>> Peter
>> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>,
>> wrote:
>>
>> Can you share the stacktrace from the logs?
>>
>> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com>
>> wrote:
>>
>>> Mike,
>>>
>>> The SSLContextService only had the Trust store configured. I think I
>>> seen that ticket before but didn’t pay attention to the fact it wasn’t
>>> merged in to the code I am running.
>>>
>>> However, I configured the service to have a KeyStore now but I am
>>> getting the same errors…
>>>
>>> Thanks,
>>>
>>> Peter
>>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>>> wrote:
>>>
>>> Peter,
>>>
>>> Are you configuring the service as a trust-only configuration? If so,
>>> that's been addressed in the 1.10 which is due for release in the near(ish)
>>> future.
>>>
>>> https://issues.apache.org/jira/browse/NIFI-6228
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>>> As a follow-up.
>>>>
>>>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>>>> specify the —cacert option giving it the self-signed root certificate.
>>>>
>>>> Of course, this isn’t using the TrustStore but I am able to use the
>>>> TrustStore if I use other ES processors… just not the
>>>> ElasticSearchClientServicesImpl.
>>>>
>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>>>> wrote:
>>>>
>>>> Hi Andy,
>>>>
>>>> thanks for your suggestions. Here is what I have tried so far (still no
>>>> luck).
>>>>
>>>> Connecting with openssl and viewing the certs it presents
>>>>
>>>> *openssl s_client -connect quickstart-es-http.es
>>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>>
>>>> If I then look inside the server cert I can find this
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>>
>>>>
>>>> If I look in to the self-signed root cert I find this:
>>>>
>>>>
>>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>>
>>>>
>>>> I now double check  my trust store to make sure the Root Cert is there.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Trust store content Your keystore contains 1 entry Alias name:
>>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>>> DigitalSignature Key_CertSign ]*
>>>>
>>>> So everything looks Ok. But when I run the
>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>>> store I still get the following output in the log.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>> ... 9 common frames omitted Caused by:
>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>> valid certification path to requested target at
>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>> ... 17 common frames omitted*
>>>>
>>>> Both the Nifi install and Elastic Search install is running in
>>>> Kubernetes. The address I am using is a service address that is backed by 3
>>>> ES instances. However, I double checked all three of the ES nodes to make
>>>> sure that they returned back the same SSL cert and they did.
>>>>
>>>> The only thing I haven't been able to figure out is how to check if
>>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>>> that it would return different SSL certs but stranger things have happened…
>>>>
>>>>
>>>>
>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>>> wrote:
>>>>
>>>> Hi Peter,
>>>>
>>>> If you can use openssl’s s_client command (example below) to connect to
>>>> the endpoint and verify that the hostname matches the certificate and that
>>>> the certificate contains a SubjectAlternativeName entry with that hostname
>>>> (see RFC 6125 [1] for more details), this should help you debug the issue.
>>>> The cause of the PKIX error is that the truststore doesn’t contain a
>>>> certificate (or certificate chain) which matches the hostname presented by
>>>> the remote endpoint. I think you understand that based on your message. The
>>>> underlying reason for this is could be one of the following:
>>>>
>>>> * the server is behind an interface which responds differently to GET
>>>> and POST/PUT requests
>>>> * there is a load-balancer which is directing the requests
>>>> coincidentally to different backend servers (one has the right cert; the
>>>> other doesn’t)
>>>> * I recall something around the addition of (some) Elastic Search
>>>> components which handled TLS in an ES client-specific manner; I remember
>>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>>> was ultimately contributed. If it’s not one of the above issues, I can
>>>> investigate further.
>>>>
>>>> Hopefully this helps.
>>>>
>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>>
>>>> s_client example:
>>>>
>>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>>> <path_to_your_CA_cert.pem>
>>>>
>>>> Andy LoPresto
>>>> alopresto@apache.org
>>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>
>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>>> wrote:
>>>>
>>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>>> flow that has the PutElasticSearchHttp component. I have setup a
>>>> SSLContextService for that component where I have specified a trust store
>>>> that has the self-signed cert from ES. I specify an https endpoint to
>>>> access Elastic Search and Im having no issues populating my Elastic Search
>>>> instance using this flow.
>>>>
>>>> I have another flow where I want to do some lookups. So I have been
>>>> using the LookupRecord processor. That one I have associated with an
>>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>>> checked this). However, when I run this second Flow I am not able to verify
>>>> the ES server's self-signed certificate.
>>>>
>>>> I check the nifi-app.log and it says:
>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>>> unable to find valid certification path to requested target
>>>>
>>>> I am a bit surprised that I am not able to verify the same server
>>>> certificate in the two different flows.
>>>>
>>>> Completely stuck on this so if anyone have any pointers please let me
>>>> know.
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>>
>>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
The certs in the TrustStore are marked as trusted.

The host name specified in the ClientServiceImpl is:
https://quickstart-es-http.es-cluster:9200

The CN field of the server certificate is:
https://quickstart-es-http.es-cluster.es.local

So at first glance it looks like the issue would be that the CN field differs from the host name. However, Im not getting an error message saying that the HostName doesn’t match the CN. According to the RFC spec first mentioned by Andy the CN field should only be used if the SubjectiveAlternativeName is empty.

The Server Cert has the SAN set to:  quickstart-es-http.es-cluster

What is really strange to me is why I can connect using the QueryElasticSearch processor with the same SSLContextService and hostname but when I use the ClientServicesImpl it will not work.

So I did some more investigation and it looks like the QueryES processor uses the okHTTP library and ClientServicesImpl uses the Elastic Search RestClient which in turn uses the HTTPClient from Apache.

There was an issue with the HTTPClient library in v4.5.2 (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t do the right hostname verification.

The version of Apache Nifi that Im using is 1.9.2 and it seems to use v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of Apache HTTPClient.

https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties

But again if it was a HostName validation issue the error message should read more like: “Certificate for <host> doesn’t match common name of certificate subject…”

On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> I'm far from a SSL/TLS expert, but let's get these out of the way:
>
> 1. Did you mark the server's cert as "trusted" when you created the trust store with keytool?
> 2. Are you sure that you're specifying the same hostname in the client service that is in the CN field in the server's cert?
>
> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use it to generate valid certificates. It's meant for NiFi, but the certs it generates should be a solid step up from self-signed certs if you're not able to access a company CA.
>
> > On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > Nope. No good. I even dump the network traffic and analyzed it in Wireshark. The ES server sends back two certificates (server + self-signed one) and both of them are present in my TrustStore. I am specifying both a TrustStore and a Keystore now but it still gives the error that it can’t find the certificate.
> > >
> > > Thanks,
> > >
> > > Peter
> > > On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > Think I might have found the issue. Will report tonight.
> > > >
> > > > Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far.
> > > >
> > > > Thanks,
> > > >
> > > > Peter
> > > > On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > Here it is:
> > > > >
> > > > >
> > > > > 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default, section=1], offset=7180, length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> > > > > org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> > > > > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
> > > > > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
> > > > > at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
> > > > > at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
> > > > > at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
> > > > > at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
> > > > > at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> > > > > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> > > > > at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> > > > > at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> > > > > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> > > > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > Caused by: org.apache.nifi.lookup.LookupFailureException: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
> > > > > at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
> > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> > > > > at com.sun.proxy.$Proxy136.lookup(Unknown Source)
> > > > > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
> > > > > ... 17 common frames omitted
> > > > > Caused by: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
> > > > > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
> > > > > ... 23 common frames omitted
> > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> > > > > at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> > > > > at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> > > > > at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> > > > > at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
> > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
> > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
> > > > > at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
> > > > > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> > > > > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> > > > > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> > > > > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> > > > > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> > > > > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> > > > > ... 1 common frames omitted
> > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > ... 9 common frames omitted
> > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > ... 17 common frames omitted
> > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> > > > > at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> > > > > at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> > > > > ... 23 common frames omitted
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Peter
> > > > > On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > > Can you share the stacktrace from the logs?
> > > > > >
> > > > > > > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > Mike,
> > > > > > > >
> > > > > > > > The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running.
> > > > > > > >
> > > > > > > > However, I configured the service to have a KeyStore now but I am getting the same errors…
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Peter
> > > > > > > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > > > > > Peter,
> > > > > > > > >
> > > > > > > > > Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
> > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/NIFI-6228
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > > > As a follow-up.
> > > > > > > > > > >
> > > > > > > > > > > On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.
> > > > > > > > > > >
> > > > > > > > > > > Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.
> > > > > > > > > > >
> > > > > > > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > > > > > > Hi Andy,
> > > > > > > > > > > >
> > > > > > > > > > > > thanks for your suggestions. Here is what I have tried so far (still no luck).
> > > > > > > > > > > >
> > > > > > > > > > > > Connecting with openssl and viewing the certs it presents
> > > > > > > > > > > >
> > > > > > > > > > > > openssl s_client -connect quickstart-es-http.es-cluster -showcerts
> > > > > > > > > > > >
> > > > > > > > > > > > If I then look inside the server cert I can find this
> > > > > > > > > > > >
> > > > > > > > > > > > Server Cert:
> > > > > > > > > > > > Issuer: OU = quickstart, CN = quickstart-http
> > > > > > > > > > > >
> > > > > > > > > > > > X509v3 Subject Alternative Name:
> > > > > > > > > > > > DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If I look in to the self-signed root cert I find this:
> > > > > > > > > > > >
> > > > > > > > > > > > Root Cert:
> > > > > > > > > > > > Subject: OU = quickstart, CN = quickstart-http
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I now double check  my trust store to make sure the Root Cert is there.
> > > > > > > > > > > >
> > > > > > > > > > > > Trust store content
> > > > > > > > > > > > Your keystore contains 1 entry
> > > > > > > > > > > >
> > > > > > > > > > > > Alias name: ca_elastic
> > > > > > > > > > > > Creation date: Oct 16, 2019
> > > > > > > > > > > > Entry type: trustedCertEntry
> > > > > > > > > > > >
> > > > > > > > > > > > Owner: CN=quickstart-http, OU=quickstart
> > > > > > > > > > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > > > > > > > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > > > > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> > > > > > > > > > > > Certificate fingerprints:
> > > > > > > > > > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > > > > > > > > > SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > > > > > > > > > SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > > > > > > > > > Signature algorithm name: SHA256withRSA
> > > > > > > > > > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > > > > > > > > > Version: 3
> > > > > > > > > > > >
> > > > > > > > > > > > Extensions:
> > > > > > > > > > > >
> > > > > > > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > > > > > > > > > BasicConstraints:[
> > > > > > > > > > > > CA:true
> > > > > > > > > > > > PathLen:2147483647
> > > > > > > > > > > > ]
> > > > > > > > > > > >
> > > > > > > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > > > > > > > > > ExtendedKeyUsages [
> > > > > > > > > > > > serverAuth
> > > > > > > > > > > > clientAuth
> > > > > > > > > > > > ]
> > > > > > > > > > > >
> > > > > > > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > > > > > > > > > KeyUsage [
> > > > > > > > > > > > DigitalSignature
> > > > > > > > > > > > Key_CertSign
> > > > > > > > > > > > ]
> > > > > > > > > > > >
> > > > > > > > > > > > So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
> > > > > > > > > > > >
> > > > > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > > > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > > > > > > ... 9 common frames omitted
> > > > > > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > > > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > > > > > > ... 17 common frames omitted
> > > > > > > > > > > >
> > > > > > > > > > > > Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
> > > > > > > > > > > >
> > > > > > > > > > > > The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > > > > > > > > > > > > Hi Peter,
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> > > > > > > > > > > > >
> > > > > > > > > > > > > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > > > > > > > > > > > > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > > > > > > > > > > > > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hopefully this helps.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > > > > > > > > > >
> > > > > > > > > > > > > s_client example:
> > > > > > > > > > > > >
> > > > > > > > > > > > > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> > > > > > > > > > > > >
> > > > > > > > > > > > > Andy LoPresto
> > > > > > > > > > > > > alopresto@apache.org
> > > > > > > > > > > > > alopresto.apache@gmail.com
> > > > > > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I check the nifi-app.log and it says:
> > > > > > > > > > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Completely stuck on this so if anyone have any pointers please let me know.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Peter
> > > > > > > > > > > > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
I'm far from a SSL/TLS expert, but let's get these out of the way:

1. Did you mark the server's cert as "trusted" when you created the trust
store with keytool?
2. Are you sure that you're specifying the same hostname in the client
service that is in the CN field in the server's cert?

FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use it
to generate valid certificates. It's meant for NiFi, but the certs it
generates should be a solid step up from self-signed certs if you're not
able to access a company CA.

On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg <pe...@gmail.com> wrote:

> Nope. No good. I even dump the network traffic and analyzed it in
> Wireshark. The ES server sends back two certificates (server + self-signed
> one) and both of them are present in my TrustStore. I am specifying both a
> TrustStore and a Keystore now but it still gives the error that it can’t
> find the certificate.
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>,
> wrote:
>
> Think I might have found the issue. Will report tonight.
>
> Mike, please don’t spend any time debugging this because I think it might
> be an issue on my side. Appreciate all the help so far.
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>,
> wrote:
>
> Here it is:
>
>
> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.LookupRecord
> LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process
> StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default,
> section=1], offset=7180,
> length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]:
> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
> coordinates {key=test} in Lookup Service
> org.apache.nifi.processor.exception.ProcessException: Failed to lookup
> coordinates {key=test} in Lookup Service
> at
> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
> at
> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
> at
> org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
> at
> org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.nifi.lookup.LookupFailureException:
> org.apache.nifi.lookup.LookupFailureException:
> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at
> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
> at
> org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
> ... 17 common frames omitted
> Caused by: org.apache.nifi.lookup.LookupFailureException:
> javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at
> org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
> at
> org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
> ... 23 common frames omitted
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> ... 1 common frames omitted
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> ... 9 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
> to find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> at
> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> at sun.security.validator.Validator.validate(Validator.java:262)
> at
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> at
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> at
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> ... 17 common frames omitted
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
> at
> sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> at
> sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> ... 23 common frames omitted
>
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>,
> wrote:
>
> Can you share the stacktrace from the logs?
>
> On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com>
> wrote:
>
>> Mike,
>>
>> The SSLContextService only had the Trust store configured. I think I seen
>> that ticket before but didn’t pay attention to the fact it wasn’t merged in
>> to the code I am running.
>>
>> However, I configured the service to have a KeyStore now but I am getting
>> the same errors…
>>
>> Thanks,
>>
>> Peter
>> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
>> wrote:
>>
>> Peter,
>>
>> Are you configuring the service as a trust-only configuration? If so,
>> that's been addressed in the 1.10 which is due for release in the near(ish)
>> future.
>>
>> https://issues.apache.org/jira/browse/NIFI-6228
>>
>> Thanks,
>>
>> Mike
>>
>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>> wrote:
>>
>>> As a follow-up.
>>>
>>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>>> specify the —cacert option giving it the self-signed root certificate.
>>>
>>> Of course, this isn’t using the TrustStore but I am able to use the
>>> TrustStore if I use other ES processors… just not the
>>> ElasticSearchClientServicesImpl.
>>>
>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>>> wrote:
>>>
>>> Hi Andy,
>>>
>>> thanks for your suggestions. Here is what I have tried so far (still no
>>> luck).
>>>
>>> Connecting with openssl and viewing the certs it presents
>>>
>>> *openssl s_client -connect quickstart-es-http.es
>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>
>>> If I then look inside the server cert I can find this
>>>
>>>
>>>
>>>
>>>
>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>
>>>
>>> If I look in to the self-signed root cert I find this:
>>>
>>>
>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>
>>>
>>> I now double check  my trust store to make sure the Root Cert is there.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Trust store content Your keystore contains 1 entry Alias name:
>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>> DigitalSignature Key_CertSign ]*
>>>
>>> So everything looks Ok. But when I run the
>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>> store I still get the following output in the log.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>> at
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>> java.security.AccessController.doPrivileged(Native Method) at
>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>> ... 9 common frames omitted Caused by:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>> valid certification path to requested target at
>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>> at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>> ... 17 common frames omitted*
>>>
>>> Both the Nifi install and Elastic Search install is running in
>>> Kubernetes. The address I am using is a service address that is backed by 3
>>> ES instances. However, I double checked all three of the ES nodes to make
>>> sure that they returned back the same SSL cert and they did.
>>>
>>> The only thing I haven't been able to figure out is how to check if
>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>> that it would return different SSL certs but stranger things have happened…
>>>
>>>
>>>
>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>> wrote:
>>>
>>> Hi Peter,
>>>
>>> If you can use openssl’s s_client command (example below) to connect to
>>> the endpoint and verify that the hostname matches the certificate and that
>>> the certificate contains a SubjectAlternativeName entry with that hostname
>>> (see RFC 6125 [1] for more details), this should help you debug the issue.
>>> The cause of the PKIX error is that the truststore doesn’t contain a
>>> certificate (or certificate chain) which matches the hostname presented by
>>> the remote endpoint. I think you understand that based on your message. The
>>> underlying reason for this is could be one of the following:
>>>
>>> * the server is behind an interface which responds differently to GET
>>> and POST/PUT requests
>>> * there is a load-balancer which is directing the requests
>>> coincidentally to different backend servers (one has the right cert; the
>>> other doesn’t)
>>> * I recall something around the addition of (some) Elastic Search
>>> components which handled TLS in an ES client-specific manner; I remember
>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>> was ultimately contributed. If it’s not one of the above issues, I can
>>> investigate further.
>>>
>>> Hopefully this helps.
>>>
>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>
>>> s_client example:
>>>
>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>> <path_to_your_CA_cert.pem>
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>> flow that has the PutElasticSearchHttp component. I have setup a
>>> SSLContextService for that component where I have specified a trust store
>>> that has the self-signed cert from ES. I specify an https endpoint to
>>> access Elastic Search and Im having no issues populating my Elastic Search
>>> instance using this flow.
>>>
>>> I have another flow where I want to do some lookups. So I have been
>>> using the LookupRecord processor. That one I have associated with an
>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>> checked this). However, when I run this second Flow I am not able to verify
>>> the ES server's self-signed certificate.
>>>
>>> I check the nifi-app.log and it says:
>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>> unable to find valid certification path to requested target
>>>
>>> I am a bit surprised that I am not able to verify the same server
>>> certificate in the two different flows.
>>>
>>> Completely stuck on this so if anyone have any pointers please let me
>>> know.
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>>
>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
Nope. No good. I even dump the network traffic and analyzed it in Wireshark. The ES server sends back two certificates (server + self-signed one) and both of them are present in my TrustStore. I am specifying both a TrustStore and a Keystore now but it still gives the error that it can’t find the certificate.

Thanks,

Peter
On Oct 18, 2019, 4:44 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> Think I might have found the issue. Will report tonight.
>
> Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far.
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > Here it is:
> >
> >
> > 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default, section=1], offset=7180, length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> > org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
> > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
> > at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
> > at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
> > at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
> > at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
> > at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> > at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> > at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: org.apache.nifi.lookup.LookupFailureException: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
> > at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> > at com.sun.proxy.$Proxy136.lookup(Unknown Source)
> > at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
> > ... 17 common frames omitted
> > Caused by: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
> > at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
> > ... 23 common frames omitted
> > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> > at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> > at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> > at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> > at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
> > at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
> > at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> > at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> > at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> > at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> > ... 1 common frames omitted
> > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > ... 9 common frames omitted
> > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > at sun.security.validator.Validator.validate(Validator.java:262)
> > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > ... 17 common frames omitted
> > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> > at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> > at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> > ... 23 common frames omitted
> >
> >
> > Thanks,
> >
> > Peter
> > On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > Can you share the stacktrace from the logs?
> > >
> > > > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:
> > > > > Mike,
> > > > >
> > > > > The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running.
> > > > >
> > > > > However, I configured the service to have a KeyStore now but I am getting the same errors…
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Peter
> > > > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > > Peter,
> > > > > >
> > > > > > Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/NIFI-6228
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mike
> > > > > >
> > > > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > As a follow-up.
> > > > > > > >
> > > > > > > > On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.
> > > > > > > >
> > > > > > > > Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.
> > > > > > > >
> > > > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > > > Hi Andy,
> > > > > > > > >
> > > > > > > > > thanks for your suggestions. Here is what I have tried so far (still no luck).
> > > > > > > > >
> > > > > > > > > Connecting with openssl and viewing the certs it presents
> > > > > > > > >
> > > > > > > > > openssl s_client -connect quickstart-es-http.es-cluster -showcerts
> > > > > > > > >
> > > > > > > > > If I then look inside the server cert I can find this
> > > > > > > > >
> > > > > > > > > Server Cert:
> > > > > > > > > Issuer: OU = quickstart, CN = quickstart-http
> > > > > > > > >
> > > > > > > > > X509v3 Subject Alternative Name:
> > > > > > > > > DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > If I look in to the self-signed root cert I find this:
> > > > > > > > >
> > > > > > > > > Root Cert:
> > > > > > > > > Subject: OU = quickstart, CN = quickstart-http
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I now double check  my trust store to make sure the Root Cert is there.
> > > > > > > > >
> > > > > > > > > Trust store content
> > > > > > > > > Your keystore contains 1 entry
> > > > > > > > >
> > > > > > > > > Alias name: ca_elastic
> > > > > > > > > Creation date: Oct 16, 2019
> > > > > > > > > Entry type: trustedCertEntry
> > > > > > > > >
> > > > > > > > > Owner: CN=quickstart-http, OU=quickstart
> > > > > > > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > > > > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> > > > > > > > > Certificate fingerprints:
> > > > > > > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > > > > > > SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > > > > > > SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > > > > > > Signature algorithm name: SHA256withRSA
> > > > > > > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > > > > > > Version: 3
> > > > > > > > >
> > > > > > > > > Extensions:
> > > > > > > > >
> > > > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > > > > > > BasicConstraints:[
> > > > > > > > > CA:true
> > > > > > > > > PathLen:2147483647
> > > > > > > > > ]
> > > > > > > > >
> > > > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > > > > > > ExtendedKeyUsages [
> > > > > > > > > serverAuth
> > > > > > > > > clientAuth
> > > > > > > > > ]
> > > > > > > > >
> > > > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > > > > > > KeyUsage [
> > > > > > > > > DigitalSignature
> > > > > > > > > Key_CertSign
> > > > > > > > > ]
> > > > > > > > >
> > > > > > > > > So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
> > > > > > > > >
> > > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > > > ... 9 common frames omitted
> > > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > > > ... 17 common frames omitted
> > > > > > > > >
> > > > > > > > > Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
> > > > > > > > >
> > > > > > > > > The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > > > > > > > > > Hi Peter,
> > > > > > > > > >
> > > > > > > > > > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> > > > > > > > > >
> > > > > > > > > > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > > > > > > > > > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > > > > > > > > > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> > > > > > > > > >
> > > > > > > > > > Hopefully this helps.
> > > > > > > > > >
> > > > > > > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > > > > > > >
> > > > > > > > > > s_client example:
> > > > > > > > > >
> > > > > > > > > > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> > > > > > > > > >
> > > > > > > > > > Andy LoPresto
> > > > > > > > > > alopresto@apache.org
> > > > > > > > > > alopresto.apache@gmail.com
> > > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > > > > > > >
> > > > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > > > > > > > > > >
> > > > > > > > > > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > > > > > > > > > >
> > > > > > > > > > > I check the nifi-app.log and it says:
> > > > > > > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > > >
> > > > > > > > > > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > > > > > > > > > >
> > > > > > > > > > > Completely stuck on this so if anyone have any pointers please let me know.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Peter
> > > > > > > > > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
Think I might have found the issue. Will report tonight.

Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far.

Thanks,

Peter
On Oct 18, 2019, 2:21 PM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> Here it is:
>
>
> 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default, section=1], offset=7180, length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
> at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
> at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
> at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
> at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
> at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
> at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.nifi.lookup.LookupFailureException: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
> at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
> at com.sun.proxy.$Proxy136.lookup(Unknown Source)
> at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
> ... 17 common frames omitted
> Caused by: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
> at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
> ... 23 common frames omitted
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
> at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
> at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
> at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
> at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
> at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
> ... 1 common frames omitted
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> ... 9 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> at sun.security.validator.Validator.validate(Validator.java:262)
> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> ... 17 common frames omitted
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
> at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
> at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
> ... 23 common frames omitted
>
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > Can you share the stacktrace from the logs?
> >
> > > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:
> > > > Mike,
> > > >
> > > > The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running.
> > > >
> > > > However, I configured the service to have a KeyStore now but I am getting the same errors…
> > > >
> > > > Thanks,
> > > >
> > > > Peter
> > > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > > Peter,
> > > > >
> > > > > Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
> > > > >
> > > > > https://issues.apache.org/jira/browse/NIFI-6228
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mike
> > > > >
> > > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > As a follow-up.
> > > > > > >
> > > > > > > On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.
> > > > > > >
> > > > > > > Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.
> > > > > > >
> > > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > > Hi Andy,
> > > > > > > >
> > > > > > > > thanks for your suggestions. Here is what I have tried so far (still no luck).
> > > > > > > >
> > > > > > > > Connecting with openssl and viewing the certs it presents
> > > > > > > >
> > > > > > > > openssl s_client -connect quickstart-es-http.es-cluster -showcerts
> > > > > > > >
> > > > > > > > If I then look inside the server cert I can find this
> > > > > > > >
> > > > > > > > Server Cert:
> > > > > > > > Issuer: OU = quickstart, CN = quickstart-http
> > > > > > > >
> > > > > > > > X509v3 Subject Alternative Name:
> > > > > > > > DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
> > > > > > > >
> > > > > > > >
> > > > > > > > If I look in to the self-signed root cert I find this:
> > > > > > > >
> > > > > > > > Root Cert:
> > > > > > > > Subject: OU = quickstart, CN = quickstart-http
> > > > > > > >
> > > > > > > >
> > > > > > > > I now double check  my trust store to make sure the Root Cert is there.
> > > > > > > >
> > > > > > > > Trust store content
> > > > > > > > Your keystore contains 1 entry
> > > > > > > >
> > > > > > > > Alias name: ca_elastic
> > > > > > > > Creation date: Oct 16, 2019
> > > > > > > > Entry type: trustedCertEntry
> > > > > > > >
> > > > > > > > Owner: CN=quickstart-http, OU=quickstart
> > > > > > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > > > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> > > > > > > > Certificate fingerprints:
> > > > > > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > > > > > SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > > > > > SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > > > > > Signature algorithm name: SHA256withRSA
> > > > > > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > > > > > Version: 3
> > > > > > > >
> > > > > > > > Extensions:
> > > > > > > >
> > > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > > > > > BasicConstraints:[
> > > > > > > > CA:true
> > > > > > > > PathLen:2147483647
> > > > > > > > ]
> > > > > > > >
> > > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > > > > > ExtendedKeyUsages [
> > > > > > > > serverAuth
> > > > > > > > clientAuth
> > > > > > > > ]
> > > > > > > >
> > > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > > > > > KeyUsage [
> > > > > > > > DigitalSignature
> > > > > > > > Key_CertSign
> > > > > > > > ]
> > > > > > > >
> > > > > > > > So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
> > > > > > > >
> > > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > > ... 9 common frames omitted
> > > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > > ... 17 common frames omitted
> > > > > > > >
> > > > > > > > Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
> > > > > > > >
> > > > > > > > The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > > > > > > > > Hi Peter,
> > > > > > > > >
> > > > > > > > > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> > > > > > > > >
> > > > > > > > > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > > > > > > > > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > > > > > > > > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> > > > > > > > >
> > > > > > > > > Hopefully this helps.
> > > > > > > > >
> > > > > > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > > > > > >
> > > > > > > > > s_client example:
> > > > > > > > >
> > > > > > > > > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> > > > > > > > >
> > > > > > > > > Andy LoPresto
> > > > > > > > > alopresto@apache.org
> > > > > > > > > alopresto.apache@gmail.com
> > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > > > > > >
> > > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > > > > > > > > >
> > > > > > > > > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > > > > > > > > >
> > > > > > > > > > I check the nifi-app.log and it says:
> > > > > > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > > >
> > > > > > > > > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > > > > > > > > >
> > > > > > > > > > Completely stuck on this so if anyone have any pointers please let me know.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Peter
> > > > > > > > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
Here it is:


2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000-0000-000065536eb2] Failed to process StandardFlowFileRecord[uuid=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1571410822200-1, container=default, section=1], offset=7180, length=20],offset=0,name=64d0d1f4-1960-4a91-9394-39edc9d6c9c7,size=20]: org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
org.apache.nifi.processor.exception.ProcessException: Failed to lookup coordinates {key=test} in Lookup Service
at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:300)
at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:68)
at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:134)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180)
at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.lookup.LookupFailureException: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:175)
at sun.reflect.GeneratedMethodAccessor746.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:87)
at com.sun.proxy.$Proxy136.lookup(Unknown Source)
at org.apache.nifi.processors.standard.LookupRecord.route(LookupRecord.java:298)
... 17 common frames omitted
Caused by: org.apache.nifi.lookup.LookupFailureException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at org.apache.nifi.elasticsearch.ElasticSearchLookupService.getByQuery(ElasticSearchLookupService.java:288)
at org.apache.nifi.elasticsearch.ElasticSearchLookupService.lookup(ElasticSearchLookupService.java:169)
... 23 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
... 9 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
... 17 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 23 common frames omitted


Thanks,

Peter
On Oct 18, 2019, 2:15 PM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> Can you share the stacktrace from the logs?
>
> > On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:
> > > Mike,
> > >
> > > The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running.
> > >
> > > However, I configured the service to have a KeyStore now but I am getting the same errors…
> > >
> > > Thanks,
> > >
> > > Peter
> > > On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> > > > Peter,
> > > >
> > > > Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
> > > >
> > > > https://issues.apache.org/jira/browse/NIFI-6228
> > > >
> > > > Thanks,
> > > >
> > > > Mike
> > > >
> > > > > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > > > > As a follow-up.
> > > > > >
> > > > > > On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.
> > > > > >
> > > > > > Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.
> > > > > >
> > > > > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > > > > Hi Andy,
> > > > > > >
> > > > > > > thanks for your suggestions. Here is what I have tried so far (still no luck).
> > > > > > >
> > > > > > > Connecting with openssl and viewing the certs it presents
> > > > > > >
> > > > > > > openssl s_client -connect quickstart-es-http.es-cluster -showcerts
> > > > > > >
> > > > > > > If I then look inside the server cert I can find this
> > > > > > >
> > > > > > > Server Cert:
> > > > > > > Issuer: OU = quickstart, CN = quickstart-http
> > > > > > >
> > > > > > > X509v3 Subject Alternative Name:
> > > > > > > DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
> > > > > > >
> > > > > > >
> > > > > > > If I look in to the self-signed root cert I find this:
> > > > > > >
> > > > > > > Root Cert:
> > > > > > > Subject: OU = quickstart, CN = quickstart-http
> > > > > > >
> > > > > > >
> > > > > > > I now double check  my trust store to make sure the Root Cert is there.
> > > > > > >
> > > > > > > Trust store content
> > > > > > > Your keystore contains 1 entry
> > > > > > >
> > > > > > > Alias name: ca_elastic
> > > > > > > Creation date: Oct 16, 2019
> > > > > > > Entry type: trustedCertEntry
> > > > > > >
> > > > > > > Owner: CN=quickstart-http, OU=quickstart
> > > > > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> > > > > > > Certificate fingerprints:
> > > > > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > > > > SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > > > > SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > > > > Signature algorithm name: SHA256withRSA
> > > > > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > > > > Version: 3
> > > > > > >
> > > > > > > Extensions:
> > > > > > >
> > > > > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > > > > BasicConstraints:[
> > > > > > > CA:true
> > > > > > > PathLen:2147483647
> > > > > > > ]
> > > > > > >
> > > > > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > > > > ExtendedKeyUsages [
> > > > > > > serverAuth
> > > > > > > clientAuth
> > > > > > > ]
> > > > > > >
> > > > > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > > > > KeyUsage [
> > > > > > > DigitalSignature
> > > > > > > Key_CertSign
> > > > > > > ]
> > > > > > >
> > > > > > > So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
> > > > > > >
> > > > > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > > > > ... 9 common frames omitted
> > > > > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > > > > ... 17 common frames omitted
> > > > > > >
> > > > > > > Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
> > > > > > >
> > > > > > > The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > > > > > > > Hi Peter,
> > > > > > > >
> > > > > > > > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> > > > > > > >
> > > > > > > > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > > > > > > > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > > > > > > > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> > > > > > > >
> > > > > > > > Hopefully this helps.
> > > > > > > >
> > > > > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > > > > >
> > > > > > > > s_client example:
> > > > > > > >
> > > > > > > > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> > > > > > > >
> > > > > > > > Andy LoPresto
> > > > > > > > alopresto@apache.org
> > > > > > > > alopresto.apache@gmail.com
> > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > > > > >
> > > > > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > > > > > > > >
> > > > > > > > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > > > > > > > >
> > > > > > > > > I check the nifi-app.log and it says:
> > > > > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > > > > >
> > > > > > > > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > > > > > > > >
> > > > > > > > > Completely stuck on this so if anyone have any pointers please let me know.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Peter
> > > > > > > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
Can you share the stacktrace from the logs?

On Fri, Oct 18, 2019 at 2:38 PM Peter Moberg <pe...@gmail.com> wrote:

> Mike,
>
> The SSLContextService only had the Trust store configured. I think I seen
> that ticket before but didn’t pay attention to the fact it wasn’t merged in
> to the code I am running.
>
> However, I configured the service to have a KeyStore now but I am getting
> the same errors…
>
> Thanks,
>
> Peter
> On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>,
> wrote:
>
> Peter,
>
> Are you configuring the service as a trust-only configuration? If so,
> that's been addressed in the 1.10 which is due for release in the near(ish)
> future.
>
> https://issues.apache.org/jira/browse/NIFI-6228
>
> Thanks,
>
> Mike
>
> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
> wrote:
>
>> As a follow-up.
>>
>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>> specify the —cacert option giving it the self-signed root certificate.
>>
>> Of course, this isn’t using the TrustStore but I am able to use the
>> TrustStore if I use other ES processors… just not the
>> ElasticSearchClientServicesImpl.
>>
>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>> wrote:
>>
>> Hi Andy,
>>
>> thanks for your suggestions. Here is what I have tried so far (still no
>> luck).
>>
>> Connecting with openssl and viewing the certs it presents
>>
>> *openssl s_client -connect quickstart-es-http.es
>> <http://quickstart-es-http.es>-cluster -showcerts*
>>
>> If I then look inside the server cert I can find this
>>
>>
>>
>>
>>
>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>> Subject Alternative Name: DNS:quickstart-es-http.es
>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>
>>
>> If I look in to the self-signed root cert I find this:
>>
>>
>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>
>>
>> I now double check  my trust store to make sure the Root Cert is there.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Trust store content Your keystore contains 1 entry Alias name:
>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>> DigitalSignature Key_CertSign ]*
>>
>> So everything looks Ok. But when I run the
>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>> store I still get the following output in the log.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>> at
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>> java.security.AccessController.doPrivileged(Native Method) at
>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>> ... 9 common frames omitted Caused by:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target at
>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>> at sun.security.validator.Validator.validate(Validator.java:262) at
>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>> at
>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>> at
>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>> at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>> ... 17 common frames omitted*
>>
>> Both the Nifi install and Elastic Search install is running in
>> Kubernetes. The address I am using is a service address that is backed by 3
>> ES instances. However, I double checked all three of the ES nodes to make
>> sure that they returned back the same SSL cert and they did.
>>
>> The only thing I haven't been able to figure out is how to check if
>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>> that it would return different SSL certs but stranger things have happened…
>>
>>
>>
>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>> wrote:
>>
>> Hi Peter,
>>
>> If you can use openssl’s s_client command (example below) to connect to
>> the endpoint and verify that the hostname matches the certificate and that
>> the certificate contains a SubjectAlternativeName entry with that hostname
>> (see RFC 6125 [1] for more details), this should help you debug the issue.
>> The cause of the PKIX error is that the truststore doesn’t contain a
>> certificate (or certificate chain) which matches the hostname presented by
>> the remote endpoint. I think you understand that based on your message. The
>> underlying reason for this is could be one of the following:
>>
>> * the server is behind an interface which responds differently to GET and
>> POST/PUT requests
>> * there is a load-balancer which is directing the requests coincidentally
>> to different backend servers (one has the right cert; the other doesn’t)
>> * I recall something around the addition of (some) Elastic Search
>> components which handled TLS in an ES client-specific manner; I remember
>> advocating for standard NiFi TLS interaction here but I am not sure what
>> was ultimately contributed. If it’s not one of the above issues, I can
>> investigate further.
>>
>> Hopefully this helps.
>>
>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>
>> s_client example:
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert
>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> <path_to_your_CA_cert.pem>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
>>
>> I have an Elastic Search cluster that is setup with SSL. It uses a
>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>> flow that has the PutElasticSearchHttp component. I have setup a
>> SSLContextService for that component where I have specified a trust store
>> that has the self-signed cert from ES. I specify an https endpoint to
>> access Elastic Search and Im having no issues populating my Elastic Search
>> instance using this flow.
>>
>> I have another flow where I want to do some lookups. So I have been using
>> the LookupRecord processor. That one I have associated with an
>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>> SSLContextService as used above. I specified the same HTTPS Url (triple
>> checked this). However, when I run this second Flow I am not able to verify
>> the ES server's self-signed certificate.
>>
>> I check the nifi-app.log and it says:
>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable to find valid certification path to requested target
>>
>> I am a bit surprised that I am not able to verify the same server
>> certificate in the two different flows.
>>
>> Completely stuck on this so if anyone have any pointers please let me
>> know.
>>
>> Thanks,
>>
>> Peter
>>
>>
>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
Mike,

The SSLContextService only had the Trust store configured. I think I seen that ticket before but didn’t pay attention to the fact it wasn’t merged in to the code I am running.

However, I configured the service to have a KeyStore now but I am getting the same errors…

Thanks,

Peter
On Oct 18, 2019, 11:39 AM -0500, Mike Thomsen <mi...@gmail.com>, wrote:
> Peter,
>
> Are you configuring the service as a trust-only configuration? If so, that's been addressed in the 1.10 which is due for release in the near(ish) future.
>
> https://issues.apache.org/jira/browse/NIFI-6228
>
> Thanks,
>
> Mike
>
> > On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com> wrote:
> > > As a follow-up.
> > >
> > > On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.
> > >
> > > Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.
> > >
> > > On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> > > > Hi Andy,
> > > >
> > > > thanks for your suggestions. Here is what I have tried so far (still no luck).
> > > >
> > > > Connecting with openssl and viewing the certs it presents
> > > >
> > > > openssl s_client -connect quickstart-es-http.es-cluster -showcerts
> > > >
> > > > If I then look inside the server cert I can find this
> > > >
> > > > Server Cert:
> > > > Issuer: OU = quickstart, CN = quickstart-http
> > > >
> > > > X509v3 Subject Alternative Name:
> > > > DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
> > > >
> > > >
> > > > If I look in to the self-signed root cert I find this:
> > > >
> > > > Root Cert:
> > > > Subject: OU = quickstart, CN = quickstart-http
> > > >
> > > >
> > > > I now double check  my trust store to make sure the Root Cert is there.
> > > >
> > > > Trust store content
> > > > Your keystore contains 1 entry
> > > >
> > > > Alias name: ca_elastic
> > > > Creation date: Oct 16, 2019
> > > > Entry type: trustedCertEntry
> > > >
> > > > Owner: CN=quickstart-http, OU=quickstart
> > > > Issuer: CN=quickstart-http, OU=quickstart
> > > > Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> > > > Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> > > > Certificate fingerprints:
> > > > MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> > > > SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> > > > SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> > > > Signature algorithm name: SHA256withRSA
> > > > Subject Public Key Algorithm: 2048-bit RSA key
> > > > Version: 3
> > > >
> > > > Extensions:
> > > >
> > > > #1: ObjectId: 2.5.29.19 Criticality=true
> > > > BasicConstraints:[
> > > > CA:true
> > > > PathLen:2147483647
> > > > ]
> > > >
> > > > #2: ObjectId: 2.5.29.37 Criticality=false
> > > > ExtendedKeyUsages [
> > > > serverAuth
> > > > clientAuth
> > > > ]
> > > >
> > > > #3: ObjectId: 2.5.29.15 Criticality=true
> > > > KeyUsage [
> > > > DigitalSignature
> > > > Key_CertSign
> > > > ]
> > > >
> > > > So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
> > > >
> > > > Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> > > > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> > > > at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> > > > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> > > > at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> > > > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> > > > at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> > > > at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> > > > ... 9 common frames omitted
> > > > Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> > > > at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> > > > at sun.security.validator.Validator.validate(Validator.java:262)
> > > > at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> > > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> > > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> > > > at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> > > > ... 17 common frames omitted
> > > >
> > > > Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
> > > >
> > > > The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
> > > >
> > > >
> > > >
> > > > On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > > > > Hi Peter,
> > > > >
> > > > > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> > > > >
> > > > > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > > > > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > > > > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> > > > >
> > > > > Hopefully this helps.
> > > > >
> > > > > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> > > > >
> > > > > s_client example:
> > > > >
> > > > > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> > > > >
> > > > > Andy LoPresto
> > > > > alopresto@apache.org
> > > > > alopresto.apache@gmail.com
> > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > > >
> > > > > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > > > > >
> > > > > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > > > > >
> > > > > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > > > > >
> > > > > > I check the nifi-app.log and it says:
> > > > > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > > > > >
> > > > > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > > > > >
> > > > > > Completely stuck on this so if anyone have any pointers please let me know.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Peter
> > > > >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Juan Pablo Gardella <ga...@gmail.com>.
I have an issue to validate, reported at before:
http://apache-nifi.1125220.n5.nabble.com/Error-instantiating-template-on-cluster-The-specified-observer-identifier-already-exists-td12973.html

I reproduced it at apache nifi 1.5.0. I will try to check at nifi 1.9.2.
The template to load is near to 50MB.

Juan

On Fri, 18 Oct 2019 at 14:13, Joe Witt <jo...@gmail.com> wrote:

> is a daily effort at this point.  i am close to pushing first rc.  have
> been watching for stability on bug fixes.
>
> On Fri, Oct 18, 2019 at 1:10 PM Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
>> Any ETA for Nifi 1.10 release?
>>
>> On Fri, 18 Oct 2019 at 13:39, Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> Peter,
>>>
>>> Are you configuring the service as a trust-only configuration? If so,
>>> that's been addressed in the 1.10 which is due for release in the near(ish)
>>> future.
>>>
>>> https://issues.apache.org/jira/browse/NIFI-6228
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>>> As a follow-up.
>>>>
>>>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>>>> specify the —cacert option giving it the self-signed root certificate.
>>>>
>>>> Of course, this isn’t using the TrustStore but I am able to use the
>>>> TrustStore if I use other ES processors… just not the
>>>> ElasticSearchClientServicesImpl.
>>>>
>>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>>>> wrote:
>>>>
>>>> Hi Andy,
>>>>
>>>> thanks for your suggestions. Here is what I have tried so far (still no
>>>> luck).
>>>>
>>>> Connecting with openssl and viewing the certs it presents
>>>>
>>>> *openssl s_client -connect quickstart-es-http.es
>>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>>
>>>> If I then look inside the server cert I can find this
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>>
>>>>
>>>> If I look in to the self-signed root cert I find this:
>>>>
>>>>
>>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>>
>>>>
>>>> I now double check  my trust store to make sure the Root Cert is there.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Trust store content Your keystore contains 1 entry Alias name:
>>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>>> DigitalSignature Key_CertSign ]*
>>>>
>>>> So everything looks Ok. But when I run the
>>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>>> store I still get the following output in the log.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>>> java.security.AccessController.doPrivileged(Native Method) at
>>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>>> ... 9 common frames omitted Caused by:
>>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>>> valid certification path to requested target at
>>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>>> at
>>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>>> at
>>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>>> ... 17 common frames omitted*
>>>>
>>>> Both the Nifi install and Elastic Search install is running in
>>>> Kubernetes. The address I am using is a service address that is backed by 3
>>>> ES instances. However, I double checked all three of the ES nodes to make
>>>> sure that they returned back the same SSL cert and they did.
>>>>
>>>> The only thing I haven't been able to figure out is how to check if
>>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>>> that it would return different SSL certs but stranger things have happened…
>>>>
>>>>
>>>>
>>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>>> wrote:
>>>>
>>>> Hi Peter,
>>>>
>>>> If you can use openssl’s s_client command (example below) to connect to
>>>> the endpoint and verify that the hostname matches the certificate and that
>>>> the certificate contains a SubjectAlternativeName entry with that hostname
>>>> (see RFC 6125 [1] for more details), this should help you debug the issue.
>>>> The cause of the PKIX error is that the truststore doesn’t contain a
>>>> certificate (or certificate chain) which matches the hostname presented by
>>>> the remote endpoint. I think you understand that based on your message. The
>>>> underlying reason for this is could be one of the following:
>>>>
>>>> * the server is behind an interface which responds differently to GET
>>>> and POST/PUT requests
>>>> * there is a load-balancer which is directing the requests
>>>> coincidentally to different backend servers (one has the right cert; the
>>>> other doesn’t)
>>>> * I recall something around the addition of (some) Elastic Search
>>>> components which handled TLS in an ES client-specific manner; I remember
>>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>>> was ultimately contributed. If it’s not one of the above issues, I can
>>>> investigate further.
>>>>
>>>> Hopefully this helps.
>>>>
>>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>>
>>>> s_client example:
>>>>
>>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>>> <path_to_your_CA_cert.pem>
>>>>
>>>> Andy LoPresto
>>>> alopresto@apache.org
>>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>
>>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>>> wrote:
>>>>
>>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>>> flow that has the PutElasticSearchHttp component. I have setup a
>>>> SSLContextService for that component where I have specified a trust store
>>>> that has the self-signed cert from ES. I specify an https endpoint to
>>>> access Elastic Search and Im having no issues populating my Elastic Search
>>>> instance using this flow.
>>>>
>>>> I have another flow where I want to do some lookups. So I have been
>>>> using the LookupRecord processor. That one I have associated with an
>>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>>> checked this). However, when I run this second Flow I am not able to verify
>>>> the ES server's self-signed certificate.
>>>>
>>>> I check the nifi-app.log and it says:
>>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>>> unable to find valid certification path to requested target
>>>>
>>>> I am a bit surprised that I am not able to verify the same server
>>>> certificate in the two different flows.
>>>>
>>>> Completely stuck on this so if anyone have any pointers please let me
>>>> know.
>>>>
>>>> Thanks,
>>>>
>>>> Peter
>>>>
>>>>
>>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Joe Witt <jo...@gmail.com>.
is a daily effort at this point.  i am close to pushing first rc.  have
been watching for stability on bug fixes.

On Fri, Oct 18, 2019 at 1:10 PM Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:

> Any ETA for Nifi 1.10 release?
>
> On Fri, 18 Oct 2019 at 13:39, Mike Thomsen <mi...@gmail.com> wrote:
>
>> Peter,
>>
>> Are you configuring the service as a trust-only configuration? If so,
>> that's been addressed in the 1.10 which is due for release in the near(ish)
>> future.
>>
>> https://issues.apache.org/jira/browse/NIFI-6228
>>
>> Thanks,
>>
>> Mike
>>
>> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
>> wrote:
>>
>>> As a follow-up.
>>>
>>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>>> specify the —cacert option giving it the self-signed root certificate.
>>>
>>> Of course, this isn’t using the TrustStore but I am able to use the
>>> TrustStore if I use other ES processors… just not the
>>> ElasticSearchClientServicesImpl.
>>>
>>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>>> wrote:
>>>
>>> Hi Andy,
>>>
>>> thanks for your suggestions. Here is what I have tried so far (still no
>>> luck).
>>>
>>> Connecting with openssl and viewing the certs it presents
>>>
>>> *openssl s_client -connect quickstart-es-http.es
>>> <http://quickstart-es-http.es>-cluster -showcerts*
>>>
>>> If I then look inside the server cert I can find this
>>>
>>>
>>>
>>>
>>>
>>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>>> Subject Alternative Name: DNS:quickstart-es-http.es
>>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>>
>>>
>>> If I look in to the self-signed root cert I find this:
>>>
>>>
>>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>>
>>>
>>> I now double check  my trust store to make sure the Root Cert is there.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Trust store content Your keystore contains 1 entry Alias name:
>>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>>> DigitalSignature Key_CertSign ]*
>>>
>>> So everything looks Ok. But when I run the
>>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>>> store I still get the following output in the log.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>>> at
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>>> java.security.AccessController.doPrivileged(Native Method) at
>>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>>> at
>>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>>> ... 9 common frames omitted Caused by:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>> valid certification path to requested target at
>>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>>> at sun.security.validator.Validator.validate(Validator.java:262) at
>>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>>> at
>>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>>> at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>>> ... 17 common frames omitted*
>>>
>>> Both the Nifi install and Elastic Search install is running in
>>> Kubernetes. The address I am using is a service address that is backed by 3
>>> ES instances. However, I double checked all three of the ES nodes to make
>>> sure that they returned back the same SSL cert and they did.
>>>
>>> The only thing I haven't been able to figure out is how to check if
>>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>>> that it would return different SSL certs but stranger things have happened…
>>>
>>>
>>>
>>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>>> wrote:
>>>
>>> Hi Peter,
>>>
>>> If you can use openssl’s s_client command (example below) to connect to
>>> the endpoint and verify that the hostname matches the certificate and that
>>> the certificate contains a SubjectAlternativeName entry with that hostname
>>> (see RFC 6125 [1] for more details), this should help you debug the issue.
>>> The cause of the PKIX error is that the truststore doesn’t contain a
>>> certificate (or certificate chain) which matches the hostname presented by
>>> the remote endpoint. I think you understand that based on your message. The
>>> underlying reason for this is could be one of the following:
>>>
>>> * the server is behind an interface which responds differently to GET
>>> and POST/PUT requests
>>> * there is a load-balancer which is directing the requests
>>> coincidentally to different backend servers (one has the right cert; the
>>> other doesn’t)
>>> * I recall something around the addition of (some) Elastic Search
>>> components which handled TLS in an ES client-specific manner; I remember
>>> advocating for standard NiFi TLS interaction here but I am not sure what
>>> was ultimately contributed. If it’s not one of the above issues, I can
>>> investigate further.
>>>
>>> Hopefully this helps.
>>>
>>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>>
>>> s_client example:
>>>
>>> $ openssl s_client -connect <host:port> -debug -state -cert
>>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>>> <path_to_your_CA_cert.pem>
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>
>>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com>
>>> wrote:
>>>
>>> I have an Elastic Search cluster that is setup with SSL. It uses a
>>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>>> flow that has the PutElasticSearchHttp component. I have setup a
>>> SSLContextService for that component where I have specified a trust store
>>> that has the self-signed cert from ES. I specify an https endpoint to
>>> access Elastic Search and Im having no issues populating my Elastic Search
>>> instance using this flow.
>>>
>>> I have another flow where I want to do some lookups. So I have been
>>> using the LookupRecord processor. That one I have associated with an
>>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>>> SSLContextService as used above. I specified the same HTTPS Url (triple
>>> checked this). However, when I run this second Flow I am not able to verify
>>> the ES server's self-signed certificate.
>>>
>>> I check the nifi-app.log and it says:
>>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>>> unable to find valid certification path to requested target
>>>
>>> I am a bit surprised that I am not able to verify the same server
>>> certificate in the two different flows.
>>>
>>> Completely stuck on this so if anyone have any pointers please let me
>>> know.
>>>
>>> Thanks,
>>>
>>> Peter
>>>
>>>
>>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Juan Pablo Gardella <ga...@gmail.com>.
Any ETA for Nifi 1.10 release?

On Fri, 18 Oct 2019 at 13:39, Mike Thomsen <mi...@gmail.com> wrote:

> Peter,
>
> Are you configuring the service as a trust-only configuration? If so,
> that's been addressed in the 1.10 which is due for release in the near(ish)
> future.
>
> https://issues.apache.org/jira/browse/NIFI-6228
>
> Thanks,
>
> Mike
>
> On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
> wrote:
>
>> As a follow-up.
>>
>> On the Nifi node I am able to do a GET to Elastic Search using curl. I
>> specify the —cacert option giving it the self-signed root certificate.
>>
>> Of course, this isn’t using the TrustStore but I am able to use the
>> TrustStore if I use other ES processors… just not the
>> ElasticSearchClientServicesImpl.
>>
>> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
>> wrote:
>>
>> Hi Andy,
>>
>> thanks for your suggestions. Here is what I have tried so far (still no
>> luck).
>>
>> Connecting with openssl and viewing the certs it presents
>>
>> *openssl s_client -connect quickstart-es-http.es
>> <http://quickstart-es-http.es>-cluster -showcerts*
>>
>> If I then look inside the server cert I can find this
>>
>>
>>
>>
>>
>> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3
>> Subject Alternative Name: DNS:quickstart-es-http.es
>> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
>> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>>
>>
>> If I look in to the self-signed root cert I find this:
>>
>>
>> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>>
>>
>> I now double check  my trust store to make sure the Root Cert is there.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Trust store content Your keystore contains 1 entry Alias name:
>> ca_elastic Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
>> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
>> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
>> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
>> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
>> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
>> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
>> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
>> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
>> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
>> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
>> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
>> DigitalSignature Key_CertSign ]*
>>
>> So everything looks Ok. But when I run the
>> ElasticSearchClientServicesImpl with a SSLContext pointing to my trust
>> store I still get the following output in the log.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine
>> problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
>> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
>> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
>> at
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
>> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
>> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
>> java.security.AccessController.doPrivileged(Native Method) at
>> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
>> ... 9 common frames omitted Caused by:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target at
>> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
>> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
>> at sun.security.validator.Validator.validate(Validator.java:262) at
>> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>> at
>> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
>> at
>> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
>> at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
>> ... 17 common frames omitted*
>>
>> Both the Nifi install and Elastic Search install is running in
>> Kubernetes. The address I am using is a service address that is backed by 3
>> ES instances. However, I double checked all three of the ES nodes to make
>> sure that they returned back the same SSL cert and they did.
>>
>> The only thing I haven't been able to figure out is how to check if
>> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
>> that it would return different SSL certs but stranger things have happened…
>>
>>
>>
>> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
>> wrote:
>>
>> Hi Peter,
>>
>> If you can use openssl’s s_client command (example below) to connect to
>> the endpoint and verify that the hostname matches the certificate and that
>> the certificate contains a SubjectAlternativeName entry with that hostname
>> (see RFC 6125 [1] for more details), this should help you debug the issue.
>> The cause of the PKIX error is that the truststore doesn’t contain a
>> certificate (or certificate chain) which matches the hostname presented by
>> the remote endpoint. I think you understand that based on your message. The
>> underlying reason for this is could be one of the following:
>>
>> * the server is behind an interface which responds differently to GET and
>> POST/PUT requests
>> * there is a load-balancer which is directing the requests coincidentally
>> to different backend servers (one has the right cert; the other doesn’t)
>> * I recall something around the addition of (some) Elastic Search
>> components which handled TLS in an ES client-specific manner; I remember
>> advocating for standard NiFi TLS interaction here but I am not sure what
>> was ultimately contributed. If it’s not one of the above issues, I can
>> investigate further.
>>
>> Hopefully this helps.
>>
>> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>>
>> s_client example:
>>
>> $ openssl s_client -connect <host:port> -debug -state -cert
>> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
>> <path_to_your_CA_cert.pem>
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
>>
>> I have an Elastic Search cluster that is setup with SSL. It uses a
>> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
>> flow that has the PutElasticSearchHttp component. I have setup a
>> SSLContextService for that component where I have specified a trust store
>> that has the self-signed cert from ES. I specify an https endpoint to
>> access Elastic Search and Im having no issues populating my Elastic Search
>> instance using this flow.
>>
>> I have another flow where I want to do some lookups. So I have been using
>> the LookupRecord processor. That one I have associated with an
>> ElasticSearchClientServiceImpl which I have setup to  point to the same
>> SSLContextService as used above. I specified the same HTTPS Url (triple
>> checked this). However, when I run this second Flow I am not able to verify
>> the ES server's self-signed certificate.
>>
>> I check the nifi-app.log and it says:
>> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
>> unable to find valid certification path to requested target
>>
>> I am a bit surprised that I am not able to verify the same server
>> certificate in the two different flows.
>>
>> Completely stuck on this so if anyone have any pointers please let me
>> know.
>>
>> Thanks,
>>
>> Peter
>>
>>
>>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Mike Thomsen <mi...@gmail.com>.
Peter,

Are you configuring the service as a trust-only configuration? If so,
that's been addressed in the 1.10 which is due for release in the near(ish)
future.

https://issues.apache.org/jira/browse/NIFI-6228

Thanks,

Mike

On Fri, Oct 18, 2019 at 11:06 AM Peter Moberg <pe...@gmail.com>
wrote:

> As a follow-up.
>
> On the Nifi node I am able to do a GET to Elastic Search using curl. I
> specify the —cacert option giving it the self-signed root certificate.
>
> Of course, this isn’t using the TrustStore but I am able to use the
> TrustStore if I use other ES processors… just not the
> ElasticSearchClientServicesImpl.
>
> On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>,
> wrote:
>
> Hi Andy,
>
> thanks for your suggestions. Here is what I have tried so far (still no
> luck).
>
> Connecting with openssl and viewing the certs it presents
>
> *openssl s_client -connect quickstart-es-http.es
> <http://quickstart-es-http.es>-cluster -showcerts*
>
> If I then look inside the server cert I can find this
>
>
>
>
>
> *Server Cert: Issuer: OU = quickstart, CN = quickstart-http X509v3 Subject
> Alternative Name: DNS:quickstart-es-http.es
> <http://quickstart-es-http.es>-cluster.es.local, DNS:quickstart-es-http,
> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster.svc,
> DNS:quickstart-es-http.es <http://quickstart-es-http.es>-cluster*
>
>
> If I look in to the self-signed root cert I find this:
>
>
> *Root Cert: Subject: OU = quickstart, CN = quickstart-http*
>
>
> I now double check  my trust store to make sure the Root Cert is there.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Trust store content Your keystore contains 1 entry Alias name: ca_elastic
> Creation date: Oct 16, 2019 Entry type: trustedCertEntry Owner:
> CN=quickstart-http, OU=quickstart Issuer: CN=quickstart-http, OU=quickstart
> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287 Valid from: Fri Oct 11
> 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020 Certificate
> fingerprints: MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0 SHA1:
> 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99 SHA256:
> B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> Signature algorithm name: SHA256withRSA Subject Public Key Algorithm:
> 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19
> Criticality=true BasicConstraints:[ CA:true PathLen:2147483647 ] #2:
> ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth
> clientAuth ] #3: ObjectId: 2.5.29.15 Criticality=true KeyUsage [
> DigitalSignature Key_CertSign ]*
>
> So everything looks Ok. But when I run the ElasticSearchClientServicesImpl
> with a SSLContext pointing to my trust store I still get the following
> output in the log.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at
> sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at
> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at
> sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at
> sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at
> sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at
> java.security.AccessController.doPrivileged(Native Method) at
> sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> at
> org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> ... 9 common frames omitted Caused by:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target at
> sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at
> sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> at sun.security.validator.Validator.validate(Validator.java:262) at
> sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> at
> sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> at
> sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> ... 17 common frames omitted*
>
> Both the Nifi install and Elastic Search install is running in Kubernetes.
> The address I am using is a service address that is backed by 3 ES
> instances. However, I double checked all three of the ES nodes to make sure
> that they returned back the same SSL cert and they did.
>
> The only thing I haven't been able to figure out is how to check if
> Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange
> that it would return different SSL certs but stranger things have happened…
>
>
>
> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>,
> wrote:
>
> Hi Peter,
>
> If you can use openssl’s s_client command (example below) to connect to
> the endpoint and verify that the hostname matches the certificate and that
> the certificate contains a SubjectAlternativeName entry with that hostname
> (see RFC 6125 [1] for more details), this should help you debug the issue.
> The cause of the PKIX error is that the truststore doesn’t contain a
> certificate (or certificate chain) which matches the hostname presented by
> the remote endpoint. I think you understand that based on your message. The
> underlying reason for this is could be one of the following:
>
> * the server is behind an interface which responds differently to GET and
> POST/PUT requests
> * there is a load-balancer which is directing the requests coincidentally
> to different backend servers (one has the right cert; the other doesn’t)
> * I recall something around the addition of (some) Elastic Search
> components which handled TLS in an ES client-specific manner; I remember
> advocating for standard NiFi TLS interaction here but I am not sure what
> was ultimately contributed. If it’s not one of the above issues, I can
> investigate further.
>
> Hopefully this helps.
>
> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>
> s_client example:
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
>
> I have an Elastic Search cluster that is setup with SSL. It uses a
> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a
> flow that has the PutElasticSearchHttp component. I have setup a
> SSLContextService for that component where I have specified a trust store
> that has the self-signed cert from ES. I specify an https endpoint to
> access Elastic Search and Im having no issues populating my Elastic Search
> instance using this flow.
>
> I have another flow where I want to do some lookups. So I have been using
> the LookupRecord processor. That one I have associated with an
> ElasticSearchClientServiceImpl which I have setup to  point to the same
> SSLContextService as used above. I specified the same HTTPS Url (triple
> checked this). However, when I run this second Flow I am not able to verify
> the ES server's self-signed certificate.
>
> I check the nifi-app.log and it says:
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
>
> I am a bit surprised that I am not able to verify the same server
> certificate in the two different flows.
>
> Completely stuck on this so if anyone have any pointers please let me know.
>
> Thanks,
>
> Peter
>
>
>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
As a follow-up.

On the Nifi node I am able to do a GET to Elastic Search using curl. I specify the —cacert option giving it the self-signed root certificate.

Of course, this isn’t using the TrustStore but I am able to use the TrustStore if I use other ES processors… just not the ElasticSearchClientServicesImpl.

On Oct 18, 2019, 12:48 AM -0500, Peter Moberg <pe...@gmail.com>, wrote:
> Hi Andy,
>
> thanks for your suggestions. Here is what I have tried so far (still no luck).
>
> Connecting with openssl and viewing the certs it presents
>
> openssl s_client -connect quickstart-es-http.es-cluster -showcerts
>
> If I then look inside the server cert I can find this
>
> Server Cert:
> Issuer: OU = quickstart, CN = quickstart-http
>
> X509v3 Subject Alternative Name:
> DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster
>
>
> If I look in to the self-signed root cert I find this:
>
> Root Cert:
> Subject: OU = quickstart, CN = quickstart-http
>
>
> I now double check  my trust store to make sure the Root Cert is there.
>
> Trust store content
> Your keystore contains 1 entry
>
> Alias name: ca_elastic
> Creation date: Oct 16, 2019
> Entry type: trustedCertEntry
>
> Owner: CN=quickstart-http, OU=quickstart
> Issuer: CN=quickstart-http, OU=quickstart
> Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
> Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
> Certificate fingerprints:
> MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
> SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
> SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
> Signature algorithm name: SHA256withRSA
> Subject Public Key Algorithm: 2048-bit RSA key
> Version: 3
>
> Extensions:
>
> #1: ObjectId: 2.5.29.19 Criticality=true
> BasicConstraints:[
> CA:true
> PathLen:2147483647
> ]
>
> #2: ObjectId: 2.5.29.37 Criticality=false
> ExtendedKeyUsages [
> serverAuth
> clientAuth
> ]
>
> #3: ObjectId: 2.5.29.15 Criticality=true
> KeyUsage [
> DigitalSignature
> Key_CertSign
> ]
>
> So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.
>
> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
> at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
> at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
> at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
> ... 9 common frames omitted
> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
> at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
> at sun.security.validator.Validator.validate(Validator.java:262)
> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
> ... 17 common frames omitted
>
> Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.
>
> The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…
>
>
>
> On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> > Hi Peter,
> >
> > If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
> >
> > * the server is behind an interface which responds differently to GET and POST/PUT requests
> > * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> > * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
> >
> > Hopefully this helps.
> >
> > [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
> >
> > s_client example:
> >
> > $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
> >
> > Andy LoPresto
> > alopresto@apache.org
> > alopresto.apache@gmail.com
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> > >
> > > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> > >
> > > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> > >
> > > I check the nifi-app.log and it says:
> > > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> > >
> > > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> > >
> > > Completely stuck on this so if anyone have any pointers please let me know.
> > >
> > > Thanks,
> > >
> > > Peter
> >

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Peter Moberg <pe...@gmail.com>.
Hi Andy,

thanks for your suggestions. Here is what I have tried so far (still no luck).

Connecting with openssl and viewing the certs it presents

openssl s_client -connect quickstart-es-http.es-cluster -showcerts

If I then look inside the server cert I can find this

Server Cert:
Issuer: OU = quickstart, CN = quickstart-http

X509v3 Subject Alternative Name:
DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster


If I look in to the self-signed root cert I find this:

Root Cert:
Subject: OU = quickstart, CN = quickstart-http


I now double check  my trust store to make sure the Root Cert is there.

Trust store content
Your keystore contains 1 entry

Alias name: ca_elastic
Creation date: Oct 16, 2019
Entry type: trustedCertEntry

Owner: CN=quickstart-http, OU=quickstart
Issuer: CN=quickstart-http, OU=quickstart
Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
Certificate fingerprints:
MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
SHA256: B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]

#2: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_CertSign
]

So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with a SSLContext pointing to my trust store I still get the following output in the log.

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
... 9 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
... 17 common frames omitted

Both the Nifi install and Elastic Search install is running in Kubernetes. The address I am using is a service address that is backed by 3 ES instances. However, I double checked all three of the ES nodes to make sure that they returned back the same SSL cert and they did.

The only thing I haven't been able to figure out is how to check if Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that it would return different SSL certs but stranger things have happened…



On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto <al...@apache.org>, wrote:
> Hi Peter,
>
> If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:
>
> * the server is behind an interface which responds differently to GET and POST/PUT requests
> * there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
> * I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further.
>
> Hopefully this helps.
>
> [1] https://tools.ietf.org/html/rfc6125#section-6.4.4
>
> s_client example:
>
> $ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> alopresto@apache.org
> alopresto.apache@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> >
> > I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> >
> > I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> >
> > I check the nifi-app.log and it says:
> > Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> >
> > I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> >
> > Completely stuck on this so if anyone have any pointers please let me know.
> >
> > Thanks,
> >
> > Peter
>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

Posted by Andy LoPresto <al...@apache.org>.
Hi Peter,

If you can use openssl’s s_client command (example below) to connect to the endpoint and verify that the hostname matches the certificate and that the certificate contains a SubjectAlternativeName entry with that hostname (see RFC 6125 [1] for more details), this should help you debug the issue. The cause of the PKIX error is that the truststore doesn’t contain a certificate (or certificate chain) which matches the hostname presented by the remote endpoint. I think you understand that based on your message. The underlying reason for this is could be one of the following:

* the server is behind an interface which responds differently to GET and POST/PUT requests
* there is a load-balancer which is directing the requests coincidentally to different backend servers (one has the right cert; the other doesn’t)
* I recall something around the addition of (some) Elastic Search components which handled TLS in an ES client-specific manner; I remember advocating for standard NiFi TLS interaction here but I am not sure what was ultimately contributed. If it’s not one of the above issues, I can investigate further. 

Hopefully this helps. 

[1] https://tools.ietf.org/html/rfc6125#section-6.4.4 <https://tools.ietf.org/html/rfc6125#section-6.4.4>

s_client example: 

$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 16, 2019, at 8:37 PM, Peter Moberg <pe...@gmail.com> wrote:
> 
> I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow.
> 
> I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to  point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate.
> 
> I check the nifi-app.log and it says:
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
> 
> I am a bit surprised that I am not able to verify the same server certificate in the two different flows.
> 
> Completely stuck on this so if anyone have any pointers please let me know.
> 
> Thanks,
> 
> Peter