You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Henk Reder <he...@yetanalytics.com> on 2018/06/05 00:34:24 UTC
Issue securing nifi with ca-server.
Hello!
My names Henk. I'm a web developer working on getting a nifi instance setup
for some integrations.
Long story short, I setup this nifi deployment configuration back when
1.5.0 was the latest version. Now that its 1.6.0 everything works!
...except one thing.
I'm currently using tls-toolkit to setup each nodes identity in the
cluster. In order to do this, I've followed the guidelines from this post
<https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/>
back
from 2016. heres a brief overview of my previously working configuration.
*CA serverside:*
<download tls-toolkit>
<setup a linux-user>
runuser -l <linux-user> -c '/home/nifica/bin/tls-toolkit.sh server -t
<secret-token> -D CN=<ca-hostname>,OU=NIFI&'
*Nifi-client side:*
<download nifi>
<download tls-toolkit>
./nifi-toolkit/bin/tls-toolkit.sh client -c <ca-hostname> -t <secret-token>
--subjectAlternativeNames '<node-ip>,<node-hostname>' -D
'CN=<node-hostname>,OU=NIFI' -T PKCS12
I expect the actual certs and json config to comeback after the call to the
tls-toolkit server. I can verify the request is going through but I get
these obtuse messages that tell me very little.
*Nifi-client side: *
2018/06/05 00:23:43 INFO [main]
org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClient:
Requesting new certificate from <ca-hostname>:8443
2018/06/05 00:23:44 INFO [main]
org.apache.nifi.toolkit.tls.service.client.TlsCertificateSigningRequestPerformer:
Requesting certificate with dn CN=<hostname>,OU=NIFI from <ca-hostname>:8443
Service client error: Received response code 500 with payload
*CA serverside: *
2018/06/04 22:55:50 WARN [qtp1108924067-13]
org.eclipse.jetty.server.HttpChannel: /
javax.servlet.ServletException: Server error
at
org.apache.nifi.toolkit.tls.service.server.TlsCertificateAuthorityServiceHandler.handle(TlsCertificateAuthorityServiceHandler.java:99)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:369)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at
org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
at
org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
2018/06/04 22:55:50 WARN [qtp1108924067-13]
org.eclipse.jetty.server.HttpChannel:
java.lang.IllegalStateException: Error already set
at
org.eclipse.jetty.server.HttpChannelState.onError(HttpChannelState.java:743)
at
org.eclipse.jetty.server.HttpChannel.handleException(HttpChannel.java:514)
at
org.eclipse.jetty.server.HttpChannelOverHttp.handleException(HttpChannelOverHttp.java:463)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:448)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at
org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
at
org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:748)
Seeing as this code was working before, my first assumption is the problem
has something to do with some new undocumented behaviors. I fooled around
with the parameters for a bit and got very similar results, and a lot of
research and digging has left me scratching my head. I know its a 500, and
the CA server is the one throwing the exception, but I can't quite figure
out what changed and why its behaving this way. There's also no clear
indicator of what's causing the issue. I can see a callback being passed
down the stack but no message hinting at bad parameters or anything like
that.
If it helps, this is the java version I'm running on my machine:
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
If you have any further questions, please don't hesitate to ask. Thanks for
your time!
Henk
Re: Issue securing nifi with ca-server.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Hey Henk!
Could you make any progress remote debugging your issue? Although not
guaranteed to help, you may try and use strace to look into what's
happening on the syscall level.
For example I've ran this command after starting the server:
strace -fp <PID_OF_JAVA_PROCESS> -s 100 2>&1 | grep -v clock_gettime | grep
-v futex
Also it can help if you strace the first invocation of the server, because
then at least we can see what classes are loaded from the stat calls and
hopefully guess where the exception actually came from.
On Tue, Jun 5, 2018 at 8:25 PM Andy LoPresto <al...@apache.org> wrote:
> Henk,
>
> I understand if you don’t want to share the value of your CA hostname, but
> can you please double check that it doesn’t contain any spaces or other
> characters that would need to be escaped? Using double quotes is usually
> recommended for these arguments, especially as the single quotes are only
> wrapping the internal shell command to be executed as the “linux-user”. I
> see in your original client log output that it appears to be successful in
> parsing, but it doesn’t hurt to be sure.
>
> Frustrating though it may be, I would recommend enabling remote debugging
> on the toolkit server application and putting a breakpoint
> at TlsCertificateAuthorityServiceHandler.java line 99 where you can catch
> the actual exception that is being thrown and examine it. Once we have more
> information (or we have sufficient input & scenario such that we can
> reproduce this issue locally), we can open a Jira to improve the error
> handling here. Sorry the tool is not more helpful.
>
> To enable remote debugging:
>
> Modify tis-toolkit.sh line 114 to look like the following:
>
> exec "${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms12m -Xmx24m}
> -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
> org.apache.nifi.toolkit.tls.TlsToolkitMain "$@"
>
> That will pause the running application until you can start debugging in
> your IDE, and then you can resume operation on the shell.
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jun 5, 2018, at 9:39 AM, Henk Reder <he...@yetanalytics.com> wrote:
>
> Hey Peter!
>
> in response to this
> <
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/raw/%3CCALj3mbA0Ny7zMH66vrMSyp+7Fax490YAjkiwV5-Vc5cB9Hjm_g@mail.gmail.com%3E/1
> >
> .
>
> Thanks for being quick on your response. I added that -c <ca-hostname>
> parameter to the CA server setup, same issue occurs.
> That "&" is actually an artifact of linux, it just means "run whatever this
> line is on its own thread." I verified that it wasn't polluting the
> arguments.
> I also removed the --dn option from the server configuration and the issue
> was consistent, response is still a 500.
>
> I'm thinking a rollback to 1.5.0 should fix this, but it would appear that
> whatever changes happened in the transition between versions broke that
> existing configuration.
>
> Any additional ideas or should I go ahead and roll it back?
>
> Thanks again,
> Henk
>
>
> On Mon, Jun 4, 2018 at 8:34 PM, Henk Reder <he...@yetanalytics.com> wrote:
>
> Hello!
>
> My names Henk. I'm a web developer working on getting a nifi instance
> setup for some integrations.
>
> Long story short, I setup this nifi deployment configuration back when
> 1.5.0 was the latest version. Now that its 1.6.0 everything works!
> ...except one thing.
>
> I'm currently using tls-toolkit to setup each nodes identity in the
> cluster. In order to do this, I've followed the guidelines from this post
> <
> https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/>
> back
> from 2016. heres a brief overview of my previously working configuration.
>
> *CA serverside:*
>
> <download tls-toolkit>
> <setup a linux-user>
> runuser -l <linux-user> -c '/home/nifica/bin/tls-toolkit.sh server -t
> <secret-token> -D CN=<ca-hostname>,OU=NIFI&'
>
> *Nifi-client side:*
>
> <download nifi>
> <download tls-toolkit>
> ./nifi-toolkit/bin/tls-toolkit.sh client -c <ca-hostname> -t
> <secret-token> --subjectAlternativeNames '<node-ip>,<node-hostname>' -D
> 'CN=<node-hostname>,OU=NIFI' -T PKCS12
>
> I expect the actual certs and json config to comeback after the call to
> the tls-toolkit server. I can verify the request is going through but I get
> these obtuse messages that tell me very little.
>
>
>
> *Nifi-client side: *
> 2018/06/05 00:23:43 INFO [main] org.apache.nifi.toolkit.tls.
> service.client.TlsCertificateAuthorityClient: Requesting new certificate
> from <ca-hostname>:8443
> 2018/06/05 00:23:44 INFO [main] org.apache.nifi.toolkit.tls.
> service.client.TlsCertificateSigningRequestPerformer: Requesting
> certificate with dn CN=<hostname>,OU=NIFI from <ca-hostname>:8443
> Service client error: Received response code 500 with payload
>
> *CA serverside: *
>
> 2018/06/04 22:55:50 WARN [qtp1108924067-13]
> org.eclipse.jetty.server.HttpChannel:
> /
> javax.servlet.ServletException: Server error
> at org.apache.nifi.toolkit.tls.service.server.
> TlsCertificateAuthorityServiceHandler.handle(
> TlsCertificateAuthorityServiceHandler.java:99)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:564)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:369)
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
> SslConnection.java:258)
> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
> SslConnection.java:147)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(
> ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.Invocable.invokePreferred(
> Invocable.java:122)
> at org.eclipse.jetty.util.thread.strategy.
> ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:201)
> at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> 2018/06/04 22:55:50 WARN [qtp1108924067-13] org.eclipse.jetty.server.
> HttpChannel:
> java.lang.IllegalStateException: Error already set
> at org.eclipse.jetty.server.HttpChannelState.onError(
> HttpChannelState.java:743)
> at org.eclipse.jetty.server.HttpChannel.handleException(
> HttpChannel.java:514)
> at org.eclipse.jetty.server.HttpChannelOverHttp.handleException(
> HttpChannelOverHttp.java:463)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:448)
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
> SslConnection.java:258)
> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
> SslConnection.java:147)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(
> ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.Invocable.invokePreferred(
> Invocable.java:122)
> at org.eclipse.jetty.util.thread.strategy.
> ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:201)
> at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Seeing as this code was working before, my first assumption is the problem
> has something to do with some new undocumented behaviors. I fooled around
> with the parameters for a bit and got very similar results, and a lot of
> research and digging has left me scratching my head. I know its a 500, and
> the CA server is the one throwing the exception, but I can't quite figure
> out what changed and why its behaving this way. There's also no clear
> indicator of what's causing the issue. I can see a callback being passed
> down the stack but no message hinting at bad parameters or anything like
> that.
>
> If it helps, this is the java version I'm running on my machine:
>
>
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
>
> If you have any further questions, please don't hesitate to ask. Thanks
> for your time!
> Henk
>
>
>
>
Re: Issue securing nifi with ca-server.
Posted by Andy LoPresto <al...@apache.org>.
Henk,
I understand if you don’t want to share the value of your CA hostname, but can you please double check that it doesn’t contain any spaces or other characters that would need to be escaped? Using double quotes is usually recommended for these arguments, especially as the single quotes are only wrapping the internal shell command to be executed as the “linux-user”. I see in your original client log output that it appears to be successful in parsing, but it doesn’t hurt to be sure.
Frustrating though it may be, I would recommend enabling remote debugging on the toolkit server application and putting a breakpoint at TlsCertificateAuthorityServiceHandler.java line 99 where you can catch the actual exception that is being thrown and examine it. Once we have more information (or we have sufficient input & scenario such that we can reproduce this issue locally), we can open a Jira to improve the error handling here. Sorry the tool is not more helpful.
To enable remote debugging:
Modify tis-toolkit.sh line 114 to look like the following:
exec "${JAVA}" -cp "${CLASSPATH}" ${JAVA_OPTS:--Xms12m -Xmx24m} -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y org.apache.nifi.toolkit.tls.TlsToolkitMain "$@"
That will pause the running application until you can start debugging in your IDE, and then you can resume operation on the shell.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 5, 2018, at 9:39 AM, Henk Reder <he...@yetanalytics.com> wrote:
>
> Hey Peter!
>
> in response to this
> <http://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/raw/%3CCALj3mbA0Ny7zMH66vrMSyp+7Fax490YAjkiwV5-Vc5cB9Hjm_g@mail.gmail.com%3E/1>
> .
>
> Thanks for being quick on your response. I added that -c <ca-hostname>
> parameter to the CA server setup, same issue occurs.
> That "&" is actually an artifact of linux, it just means "run whatever this
> line is on its own thread." I verified that it wasn't polluting the
> arguments.
> I also removed the --dn option from the server configuration and the issue
> was consistent, response is still a 500.
>
> I'm thinking a rollback to 1.5.0 should fix this, but it would appear that
> whatever changes happened in the transition between versions broke that
> existing configuration.
>
> Any additional ideas or should I go ahead and roll it back?
>
> Thanks again,
> Henk
>
>
> On Mon, Jun 4, 2018 at 8:34 PM, Henk Reder <he...@yetanalytics.com> wrote:
>
>> Hello!
>>
>> My names Henk. I'm a web developer working on getting a nifi instance
>> setup for some integrations.
>>
>> Long story short, I setup this nifi deployment configuration back when
>> 1.5.0 was the latest version. Now that its 1.6.0 everything works!
>> ...except one thing.
>>
>> I'm currently using tls-toolkit to setup each nodes identity in the
>> cluster. In order to do this, I've followed the guidelines from this post
>> <https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/> back
>> from 2016. heres a brief overview of my previously working configuration.
>>
>> *CA serverside:*
>>
>> <download tls-toolkit>
>> <setup a linux-user>
>> runuser -l <linux-user> -c '/home/nifica/bin/tls-toolkit.sh server -t
>> <secret-token> -D CN=<ca-hostname>,OU=NIFI&'
>>
>> *Nifi-client side:*
>>
>> <download nifi>
>> <download tls-toolkit>
>> ./nifi-toolkit/bin/tls-toolkit.sh client -c <ca-hostname> -t
>> <secret-token> --subjectAlternativeNames '<node-ip>,<node-hostname>' -D
>> 'CN=<node-hostname>,OU=NIFI' -T PKCS12
>>
>> I expect the actual certs and json config to comeback after the call to
>> the tls-toolkit server. I can verify the request is going through but I get
>> these obtuse messages that tell me very little.
>>
>>
>>
>> *Nifi-client side: *
>> 2018/06/05 00:23:43 INFO [main] org.apache.nifi.toolkit.tls.
>> service.client.TlsCertificateAuthorityClient: Requesting new certificate
>> from <ca-hostname>:8443
>> 2018/06/05 00:23:44 INFO [main] org.apache.nifi.toolkit.tls.
>> service.client.TlsCertificateSigningRequestPerformer: Requesting
>> certificate with dn CN=<hostname>,OU=NIFI from <ca-hostname>:8443
>> Service client error: Received response code 500 with payload
>>
>> *CA serverside: *
>>
>> 2018/06/04 22:55:50 WARN [qtp1108924067-13] org.eclipse.jetty.server.HttpChannel:
>> /
>> javax.servlet.ServletException: Server error
>> at org.apache.nifi.toolkit.tls.service.server.
>> TlsCertificateAuthorityServiceHandler.handle(
>> TlsCertificateAuthorityServiceHandler.java:99)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:132)
>> at org.eclipse.jetty.server.Server.handle(Server.java:564)
>> at org.eclipse.jetty.server.HttpChannel.handle(
>> HttpChannel.java:369)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:251)
>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
>> AbstractConnection.java:279)
>> at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:110)
>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
>> SslConnection.java:258)
>> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
>> SslConnection.java:147)
>> at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:110)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(
>> ChannelEndPoint.java:124)
>> at org.eclipse.jetty.util.thread.Invocable.invokePreferred(
>> Invocable.java:122)
>> at org.eclipse.jetty.util.thread.strategy.
>> ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> produceConsume(ExecuteProduceConsume.java:201)
>> at org.eclipse.jetty.util.thread.strategy.
>> ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:672)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:590)
>> at java.lang.Thread.run(Thread.java:748)
>> 2018/06/04 22:55:50 WARN [qtp1108924067-13] org.eclipse.jetty.server.
>> HttpChannel:
>> java.lang.IllegalStateException: Error already set
>> at org.eclipse.jetty.server.HttpChannelState.onError(
>> HttpChannelState.java:743)
>> at org.eclipse.jetty.server.HttpChannel.handleException(
>> HttpChannel.java:514)
>> at org.eclipse.jetty.server.HttpChannelOverHttp.handleException(
>> HttpChannelOverHttp.java:463)
>> at org.eclipse.jetty.server.HttpChannel.handle(
>> HttpChannel.java:448)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:251)
>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
>> AbstractConnection.java:279)
>> at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:110)
>> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
>> SslConnection.java:258)
>> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
>> SslConnection.java:147)
>> at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:110)
>> at org.eclipse.jetty.io.ChannelEndPoint$2.run(
>> ChannelEndPoint.java:124)
>> at org.eclipse.jetty.util.thread.Invocable.invokePreferred(
>> Invocable.java:122)
>> at org.eclipse.jetty.util.thread.strategy.
>> ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
>> produceConsume(ExecuteProduceConsume.java:201)
>> at org.eclipse.jetty.util.thread.strategy.
>> ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:672)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:590)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> Seeing as this code was working before, my first assumption is the problem
>> has something to do with some new undocumented behaviors. I fooled around
>> with the parameters for a bit and got very similar results, and a lot of
>> research and digging has left me scratching my head. I know its a 500, and
>> the CA server is the one throwing the exception, but I can't quite figure
>> out what changed and why its behaving this way. There's also no clear
>> indicator of what's causing the issue. I can see a callback being passed
>> down the stack but no message hinting at bad parameters or anything like
>> that.
>>
>> If it helps, this is the java version I'm running on my machine:
>>
>>
>> openjdk version "1.8.0_171"
>> OpenJDK Runtime Environment (build 1.8.0_171-b10)
>> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
>>
>> If you have any further questions, please don't hesitate to ask. Thanks
>> for your time!
>> Henk
>>
>>
Re: Issue securing nifi with ca-server.
Posted by Henk Reder <he...@yetanalytics.com>.
Hey Peter!
in response to this
<http://mail-archives.apache.org/mod_mbox/nifi-dev/201806.mbox/raw/%3CCALj3mbA0Ny7zMH66vrMSyp+7Fax490YAjkiwV5-Vc5cB9Hjm_g@mail.gmail.com%3E/1>
.
Thanks for being quick on your response. I added that -c <ca-hostname>
parameter to the CA server setup, same issue occurs.
That "&" is actually an artifact of linux, it just means "run whatever this
line is on its own thread." I verified that it wasn't polluting the
arguments.
I also removed the --dn option from the server configuration and the issue
was consistent, response is still a 500.
I'm thinking a rollback to 1.5.0 should fix this, but it would appear that
whatever changes happened in the transition between versions broke that
existing configuration.
Any additional ideas or should I go ahead and roll it back?
Thanks again,
Henk
On Mon, Jun 4, 2018 at 8:34 PM, Henk Reder <he...@yetanalytics.com> wrote:
> Hello!
>
> My names Henk. I'm a web developer working on getting a nifi instance
> setup for some integrations.
>
> Long story short, I setup this nifi deployment configuration back when
> 1.5.0 was the latest version. Now that its 1.6.0 everything works!
> ...except one thing.
>
> I'm currently using tls-toolkit to setup each nodes identity in the
> cluster. In order to do this, I've followed the guidelines from this post
> <https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-secured-cluster-setup/> back
> from 2016. heres a brief overview of my previously working configuration.
>
> *CA serverside:*
>
> <download tls-toolkit>
> <setup a linux-user>
> runuser -l <linux-user> -c '/home/nifica/bin/tls-toolkit.sh server -t
> <secret-token> -D CN=<ca-hostname>,OU=NIFI&'
>
> *Nifi-client side:*
>
> <download nifi>
> <download tls-toolkit>
> ./nifi-toolkit/bin/tls-toolkit.sh client -c <ca-hostname> -t
> <secret-token> --subjectAlternativeNames '<node-ip>,<node-hostname>' -D
> 'CN=<node-hostname>,OU=NIFI' -T PKCS12
>
> I expect the actual certs and json config to comeback after the call to
> the tls-toolkit server. I can verify the request is going through but I get
> these obtuse messages that tell me very little.
>
>
>
> *Nifi-client side: *
> 2018/06/05 00:23:43 INFO [main] org.apache.nifi.toolkit.tls.
> service.client.TlsCertificateAuthorityClient: Requesting new certificate
> from <ca-hostname>:8443
> 2018/06/05 00:23:44 INFO [main] org.apache.nifi.toolkit.tls.
> service.client.TlsCertificateSigningRequestPerformer: Requesting
> certificate with dn CN=<hostname>,OU=NIFI from <ca-hostname>:8443
> Service client error: Received response code 500 with payload
>
> *CA serverside: *
>
> 2018/06/04 22:55:50 WARN [qtp1108924067-13] org.eclipse.jetty.server.HttpChannel:
> /
> javax.servlet.ServletException: Server error
> at org.apache.nifi.toolkit.tls.service.server.
> TlsCertificateAuthorityServiceHandler.handle(
> TlsCertificateAuthorityServiceHandler.java:99)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:564)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:369)
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
> SslConnection.java:258)
> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
> SslConnection.java:147)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(
> ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.Invocable.invokePreferred(
> Invocable.java:122)
> at org.eclipse.jetty.util.thread.strategy.
> ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:201)
> at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> 2018/06/04 22:55:50 WARN [qtp1108924067-13] org.eclipse.jetty.server.
> HttpChannel:
> java.lang.IllegalStateException: Error already set
> at org.eclipse.jetty.server.HttpChannelState.onError(
> HttpChannelState.java:743)
> at org.eclipse.jetty.server.HttpChannel.handleException(
> HttpChannel.java:514)
> at org.eclipse.jetty.server.HttpChannelOverHttp.handleException(
> HttpChannelOverHttp.java:463)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:448)
> at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
> SslConnection.java:258)
> at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
> SslConnection.java:147)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(
> ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.Invocable.invokePreferred(
> Invocable.java:122)
> at org.eclipse.jetty.util.thread.strategy.
> ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58)
> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:201)
> at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:133)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Seeing as this code was working before, my first assumption is the problem
> has something to do with some new undocumented behaviors. I fooled around
> with the parameters for a bit and got very similar results, and a lot of
> research and digging has left me scratching my head. I know its a 500, and
> the CA server is the one throwing the exception, but I can't quite figure
> out what changed and why its behaving this way. There's also no clear
> indicator of what's causing the issue. I can see a callback being passed
> down the stack but no message hinting at bad parameters or anything like
> that.
>
> If it helps, this is the java version I'm running on my machine:
>
>
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
>
> If you have any further questions, please don't hesitate to ask. Thanks
> for your time!
> Henk
>
>
Re: Issue securing nifi with ca-server.
Posted by Peter Wilcsinszky <pe...@gmail.com>.
Hey Henk!
On Tue, Jun 5, 2018 at 2:34 AM, Henk Reder <he...@yetanalytics.com> wrote:
> Hello!
>
> My names Henk. I'm a web developer working on getting a nifi instance setup
> for some integrations.
>
> Long story short, I setup this nifi deployment configuration back when
> 1.5.0 was the latest version. Now that its 1.6.0 everything works!
> ...except one thing.
>
> I'm currently using tls-toolkit to setup each nodes identity in the
> cluster. In order to do this, I've followed the guidelines from this post
> <https://pierrevillard.com/2016/11/29/apache-nifi-1-1-0-
> secured-cluster-setup/>
> back
> from 2016. heres a brief overview of my previously working configuration.
>
> *CA serverside:*
>
> <download tls-toolkit>
> <setup a linux-user>
> runuser -l <linux-user> -c '/home/nifica/bin/tls-toolkit.sh server -t
> <secret-token> -D CN=<ca-hostname>,OU=NIFI&'
>
You are missing -c <ca-hostname> which defaults to localhost if not
provided:
-c,--certificateAuthorityHostname <arg> Hostname of NiFi Certificate
Authority (default: localhost)
Also I beleive you don't have to set -D for the default case, because it
will use just what you would like to set by default:
-D,--dn <arg> The dn to use for the CA
certificate (default: CN=YOUR_CA_HOSTNAME,OU=NIFI)
Finally: is that "&" just a copy paste error in your command, right?
>
> *Nifi-client side:*
>
> <download nifi>
> <download tls-toolkit>
> ./nifi-toolkit/bin/tls-toolkit.sh client -c <ca-hostname> -t
> <secret-token>
> --subjectAlternativeNames '<node-ip>,<node-hostname>' -D
> 'CN=<node-hostname>,OU=NIFI' -T PKCS12
>
> I expect the actual certs and json config to comeback after the call to the
> tls-toolkit server. I can verify the request is going through but I get
> these obtuse messages that tell me very little.
>
>
>
> *Nifi-client side: *
> 2018/06/05 00:23:43 INFO [main]
> org.apache.nifi.toolkit.tls.service.client.TlsCertificateAuthorityClient:
> Requesting new certificate from <ca-hostname>:8443
> 2018/06/05 00:23:44 INFO [main]
> org.apache.nifi.toolkit.tls.service.client.TlsCertificateSigningRequestPe
> rformer:
> Requesting certificate with dn CN=<hostname>,OU=NIFI from
> <ca-hostname>:8443
> Service client error: Received response code 500 with payload
>
> *CA serverside: *
>
> 2018/06/04 22:55:50 WARN [qtp1108924067-13]
> org.eclipse.jetty.server.HttpChannel: /
> javax.servlet.ServletException: Server error
> at
> org.apache.nifi.toolkit.tls.service.server.TlsCertificateAuthorityService
> Handler.handle(TlsCertificateAuthorityServiceHandler.java:99)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:564)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:369)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
> at
> org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
> at
> org.eclipse.jetty.util.thread.Invocable.invokePreferred(
> Invocable.java:122)
> at
> org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(
> ExecutingExecutionStrategy.java:58)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:201)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:133)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
> 2018/06/04 22:55:50 WARN [qtp1108924067-13]
> org.eclipse.jetty.server.HttpChannel:
> java.lang.IllegalStateException: Error already set
> at
> org.eclipse.jetty.server.HttpChannelState.onError(
> HttpChannelState.java:743)
> at
> org.eclipse.jetty.server.HttpChannel.handleException(HttpChannel.java:514)
> at
> org.eclipse.jetty.server.HttpChannelOverHttp.handleException(
> HttpChannelOverHttp.java:463)
> at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:448)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
> at
> org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
> at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> at
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
> at
> org.eclipse.jetty.util.thread.Invocable.invokePreferred(
> Invocable.java:122)
> at
> org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(
> ExecutingExecutionStrategy.java:58)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:201)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:133)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> at java.lang.Thread.run(Thread.java:748)
>
>
> Seeing as this code was working before, my first assumption is the problem
> has something to do with some new undocumented behaviors. I fooled around
> with the parameters for a bit and got very similar results, and a lot of
> research and digging has left me scratching my head. I know its a 500, and
> the CA server is the one throwing the exception, but I can't quite figure
> out what changed and why its behaving this way. There's also no clear
> indicator of what's causing the issue. I can see a callback being passed
> down the stack but no message hinting at bad parameters or anything like
> that.
>
> If it helps, this is the java version I'm running on my machine:
>
>
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
>
> If you have any further questions, please don't hesitate to ask. Thanks for
> your time!
> Henk
>