Re: InvokeHTTP processor loses SSL Context after NiFi restart

[Joe Witt <jo...@gmail.com>]

[moved thread to dev as it is in pure dev mode now]

In nifi-framework-core you can follow the loading cycle from:
- StandardFlowSynchronizer
- ControllerServiceLoader
- FlowController
- StandardControllerServiceProvider

In StandardControllerServiceProvider you can see it spawns threads to
start controller services in parallel whenever possible.  The original
thread which initiated all that continues on to load processors.  The
lifecycle of starting processors protects the case exposed.  So I do
believe if in InvokeHTTP the 'createSSLContext' had not been cached or
if just the service itself was cached this would have been ok.



On Thu, Aug 6, 2015 at 1:06 PM, Adam Taft <ad...@adamtaft.com> wrote:
> Good thought, that seems reasonable.  If that's the case, then the "old
> style" should still be viable, which is good.  This would just be a one-off
> lifecycle issue.
>
> On Thu, Aug 6, 2015 at 1:03 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Adam,
>>
>> I *think* (though odds of being correctly shortly are non zero) that
>> the issue was in InvokeHTTP trying to acquire and cache a reference to
>> the controller service at a time when it might not have been available
>> yet (onPropertyModified during startup and setting properties).
>>
>> Will take a look through the startup lifecycle to verify that though.
>>
>> Thanks
>> Joe
>>
>> On Thu, Aug 6, 2015 at 12:50 PM, Adam Taft <ad...@adamtaft.com> wrote:
>> > You might want to investigate this to see if it's related to the changes
>> > with Controller Services in general.  If the newish controller service
>> > API
>> > is causing trouble with the old style reference, undoubtedly this will
>> > hit
>> > other users with old processors if/when they upgrade to Apache NiFi.
>> >
>> > Or maybe the answer is, you shouldn't use the old style anymore at all,
>> > which is also acceptable.  But if backwards compatibility is important,
>> > it
>> > might make sense to figure out "why" this stopped working recently (when
>> > clearly that style has worked for a long time).
>> >
>> > Adam
>> >
>> >
>> > On Thu, Aug 6, 2015 at 12:26 PM, Mark Payne <ma...@hotmail.com>
>> > wrote:
>> >>
>> >> Wow, let me try that again. John, I created the following ticket:
>> >> https://issues.apache.org/jira/browse/NIFI-825.  I was able to get the
>> >> same
>> >> error message using InvokeHTTP to do a GET against NiFi itself.
>> >> Interestingly, it always would fail for me, though, not just after a
>> >> restart. I updated the part of the code that Adam was suggesting may be
>> >> problematic, and it is now working well for me. This will be in our
>> >> next
>> >> release, version 0.3.0. Thanks -Mark > From: markap14@hotmail.com > To:
>> >> users@nifi.apache.org > Subject: RE: InvokeHTTP processor loses SSL
>> >> Context
>> >> after NiFi restart > Date: Thu, 6 Aug 2015 11:22:24 -0500 > > John,
>> >>
>> >> I created the following ticket:
>> >> https://issues.apache.org/jira/browse/NIFI-825
>> >>
>> >> I was able to get the same error message using InvokeHTTP to do a GET
>> >> against NiFi itself. Interestingly, it always would fail for me,
>> >> though, not
>> >> just after a restart. I updated the part of the code that Adam was
>> >> suggesting may be problematic, and it is now working well for me. This
>> >> will
>> >> be in our 0.3.0 release.
>> >>
>> >> Thanks
>> >> -Mark
>> >>
>> >> ----------------------------------------
>> >> > Date: Thu, 6 Aug 2015 09:27:26 -0500
>> >> > From: john.titus@trustedconcepts.com
>> >> > To: users@nifi.apache.org
>> >> > Subject: Re: InvokeHTTP processor loses SSL Context after NiFi
>> >> > restart
>> >> >
>> >> > I believe this is it. Not sure how useful it is. If there's something
>> >> > else I should search the log for, please let me know.
>> >> >
>> >> > 2015-08-06 14:23:06,727 ERROR [Timer-Driven Process Thread-6]
>> >> > o.a.nifi.processors.standard.InvokeHTTP
>> >> > javax.net.ssl.SSLHandshakeException:
>> >> > sun.security.validator.ValidatorException: PKIX path building failed:
>> >> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
>> >> > find
>> >> > valid certification path to requested target
>> >> > at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1937)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1050)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1282)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > org.apache.nifi.processors.standard.InvokeHTTP$Transaction.sendRequest(InvokeHTTP.java:434)
>> >> > ~[nifi-standard-processors-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.processors.standard.InvokeHTTP$Transaction.process(InvokeHTTP.java:356)
>> >> > ~[nifi-standard-processors-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:148)
>> >> > [nifi-standard-processors-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> >> > [nifi-api-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
>> >> > [nifi-framework-core-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
>> >> > [nifi-framework-core-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
>> >> > [nifi-framework-core-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>> >> > [nifi-framework-core-0.1.0-incubating.jar:0.1.0-incubating]
>> >> > at
>> >> >
>> >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >> > [na:1.8.0_45]
>> >> > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> > [na:1.8.0_45]
>> >> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> > [na:1.8.0_45]
>> >> > at
>> >> >
>> >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> > [na:1.8.0_45]
>> >> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
>> >> > Caused by: sun.security.validator.ValidatorException: PKIX path
>> >> > building
>> >> > failed: sun.security.provider.certpath.SunCertPathBuilderException:
>> >> > unable
>> >> > to find valid certification path to requested target
>> >> > at
>> >> > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
>> >> > ~[na:1.8.0_45]
>> >> > at sun.security.validator.Validator.validate(Validator.java:260)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1460)
>> >> > ~[na:1.8.0_45]
>> >> > ... 27 common frames omitted
>> >> > Caused by:
>> >> > sun.security.provider.certpath.SunCertPathBuilderException:
>> >> > unable to find valid certification path to requested target
>> >> > at
>> >> >
>> >> > sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> >
>> >> > sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
>> >> > ~[na:1.8.0_45]
>> >> > at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
>> >> > ~[na:1.8.0_45]
>> >> > at
>> >> > sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
>> >> > ~[na:1.8.0_45]
>> >> > ... 33 common frames omitted
>> >> >
>> >> >
>> >> > ----- Original Message -----
>> >> > From: "Adam Taft" <ad...@adamtaft.com>
>> >> > To: users@nifi.apache.org
>> >> > Sent: Thursday, August 6, 2015 10:13:07 AM
>> >> > Subject: Re: InvokeHTTP processor loses SSL Context after NiFi
>> >> > restart
>> >> >
>> >> >
>> >> >
>> >> > This might be related to the "old" way of finding the SSLContext from
>> >> > the controller service. I believe InvokeHTTP doesn't use the updated
>> >> > asControllerService() style. Might be where this is breaking down.
>> >> > Definitely a stack trace would help.
>> >> >
>> >> >
>> >> > Adam
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Aug 6, 2015 at 10:01 AM, Joe Witt < joe.witt@gmail.com >
>> >> > wrote:
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > I would say not a known issue. Can you possibly provide the stack
>> >> > trace/logs?
>> >> >
>> >> > That sounds super annoying though. We def want to fix.
>> >> >
>> >> > Thanks
>> >> > Joe
>> >> >
>> >> >
>> >> > On Aug 6, 2015 9:51 AM, "John Titus" < john.titus@trustedconcepts.com
>> >> > >
>> >> > wrote:
>> >> >
>> >> >
>> >> > Several of our flows use InvokeHTTP with a "SSL Context Service". If
>> >> > we
>> >> > need to restart NiFi, those processors start throwing errors. To fix
>> >> > it, we
>> >> > have to modify the config to have no SSL context, Apply the changes,
>> >> > then
>> >> > modify the config back to the correct SSL context.
>> >> >
>> >> > Is this a known issue, or do we perhaps some config wrong somewhere?
>> >> >
>> >> > John
>> >> >
>> >
>> >
>
>