You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Tyler Lubeck <ty...@tylerlubeck.com> on 2019/12/02 18:54:13 UTC
SSLAuthentication Exception JMX
Hi!
I ran into an issue over the weekend where our automated system updated
some of our brokers with a certificate signed by a new certificate
authority but didn't update the truststore on the other brokers. I've
dumped the stacktrace in a gist
<https://gist.github.com/TylerLubeck/a734514d8a2e9936f35e223f90f21307> for
context.
From what I've been able to tell, this error only shows up in logs. It'd be
useful to have this reported via other metrics as well so I can detect the
problem sooner next time around.
I *think* it'll be something like updating the SSLTransportLayer
<https://github.com/apache/kafka/blob/be58580e14be93618f11e609389ff6bb16317702/clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java#L50>
to
have a counter of number of failures and then registering itself as an
mbean via CoreUtils
<https://github.com/apache/kafka/blob/33d06082117d971cdcddd4f01392006b543f3c01/core/src/main/scala/kafka/utils/CoreUtils.scala#L140>.
That said, I'm new to Java development and have only touched JMX from the
'oh neat, metrics' side of things.
Is this worth pursuing? Is this roughly the right track? I'm more than
happy to add this in but I'm not entirely sure how to start.
Thanks,
Tyler Lubeck
(813) 469 - 1499
www.TylerLubeck.com <http://www.tylerlubeck.com/>
www.linkedin.com/in/tylerlubeck/
Re: SSLAuthentication Exception JMX
Posted by Colin McCabe <cm...@apache.org>.
Hi Tyler,
One way to find the metrics exported by Kafka is to use a tool like jconsole.
best,
Colin
On Wed, Dec 4, 2019, at 13:30, Tyler Lubeck wrote:
> Wow, thank you. That covers everything I need. If I can take up a bit more
> of your time, can you tell me how you found that?
>
> Thanks,
> Tyler Lubeck
> (813) 469 - 1499
> www.TylerLubeck.com <http://www.tylerlubeck.com/>
> www.linkedin.com/in/tylerlubeck/
>
>
>
> On Tue, Dec 3, 2019 at 3:57 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi Tyler,
> >
> > We have metrics in Selector for successful and failed authentication. On
> > the broker side, we have mbeans
> > `
> >
> > kafka.server:type=socket-server-metrics,listener=<listenerName>,networkProcessor=<processorNum>`
> >
> > These have attributes failed-authentication-rate,
> > failed-authentication-total etc. There are similar metrics on clients too.
> > Perhaps these give you what you are looking for?
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Mon, Dec 2, 2019 at 6:54 PM Tyler Lubeck <ty...@tylerlubeck.com> wrote:
> >
> > > Hi!
> > >
> > > I ran into an issue over the weekend where our automated system updated
> > > some of our brokers with a certificate signed by a new certificate
> > > authority but didn't update the truststore on the other brokers. I've
> > > dumped the stacktrace in a gist
> > > <https://gist.github.com/TylerLubeck/a734514d8a2e9936f35e223f90f21307>
> > for
> > > context.
> > >
> > > From what I've been able to tell, this error only shows up in logs. It'd
> > be
> > > useful to have this reported via other metrics as well so I can detect
> > the
> > > problem sooner next time around.
> > >
> > > I *think* it'll be something like updating the SSLTransportLayer
> > > <
> > >
> > https://github.com/apache/kafka/blob/be58580e14be93618f11e609389ff6bb16317702/clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java#L50
> > > >
> > > to
> > > have a counter of number of failures and then registering itself as an
> > > mbean via CoreUtils
> > > <
> > >
> > https://github.com/apache/kafka/blob/33d06082117d971cdcddd4f01392006b543f3c01/core/src/main/scala/kafka/utils/CoreUtils.scala#L140
> > > >.
> > > That said, I'm new to Java development and have only touched JMX from the
> > > 'oh neat, metrics' side of things.
> > >
> > > Is this worth pursuing? Is this roughly the right track? I'm more than
> > > happy to add this in but I'm not entirely sure how to start.
> > >
> > > Thanks,
> > > Tyler Lubeck
> > > (813) 469 - 1499
> > > www.TylerLubeck.com <http://www.tylerlubeck.com/>
> > > www.linkedin.com/in/tylerlubeck/
> > >
> >
>
Re: SSLAuthentication Exception JMX
Posted by Tyler Lubeck <ty...@tylerlubeck.com>.
Wow, thank you. That covers everything I need. If I can take up a bit more
of your time, can you tell me how you found that?
Thanks,
Tyler Lubeck
(813) 469 - 1499
www.TylerLubeck.com <http://www.tylerlubeck.com/>
www.linkedin.com/in/tylerlubeck/
On Tue, Dec 3, 2019 at 3:57 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Tyler,
>
> We have metrics in Selector for successful and failed authentication. On
> the broker side, we have mbeans
> `
>
> kafka.server:type=socket-server-metrics,listener=<listenerName>,networkProcessor=<processorNum>`
>
> These have attributes failed-authentication-rate,
> failed-authentication-total etc. There are similar metrics on clients too.
> Perhaps these give you what you are looking for?
>
> Regards,
>
> Rajini
>
>
> On Mon, Dec 2, 2019 at 6:54 PM Tyler Lubeck <ty...@tylerlubeck.com> wrote:
>
> > Hi!
> >
> > I ran into an issue over the weekend where our automated system updated
> > some of our brokers with a certificate signed by a new certificate
> > authority but didn't update the truststore on the other brokers. I've
> > dumped the stacktrace in a gist
> > <https://gist.github.com/TylerLubeck/a734514d8a2e9936f35e223f90f21307>
> for
> > context.
> >
> > From what I've been able to tell, this error only shows up in logs. It'd
> be
> > useful to have this reported via other metrics as well so I can detect
> the
> > problem sooner next time around.
> >
> > I *think* it'll be something like updating the SSLTransportLayer
> > <
> >
> https://github.com/apache/kafka/blob/be58580e14be93618f11e609389ff6bb16317702/clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java#L50
> > >
> > to
> > have a counter of number of failures and then registering itself as an
> > mbean via CoreUtils
> > <
> >
> https://github.com/apache/kafka/blob/33d06082117d971cdcddd4f01392006b543f3c01/core/src/main/scala/kafka/utils/CoreUtils.scala#L140
> > >.
> > That said, I'm new to Java development and have only touched JMX from the
> > 'oh neat, metrics' side of things.
> >
> > Is this worth pursuing? Is this roughly the right track? I'm more than
> > happy to add this in but I'm not entirely sure how to start.
> >
> > Thanks,
> > Tyler Lubeck
> > (813) 469 - 1499
> > www.TylerLubeck.com <http://www.tylerlubeck.com/>
> > www.linkedin.com/in/tylerlubeck/
> >
>
Re: SSLAuthentication Exception JMX
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Tyler,
We have metrics in Selector for successful and failed authentication. On
the broker side, we have mbeans
`
kafka.server:type=socket-server-metrics,listener=<listenerName>,networkProcessor=<processorNum>`
These have attributes failed-authentication-rate,
failed-authentication-total etc. There are similar metrics on clients too.
Perhaps these give you what you are looking for?
Regards,
Rajini
On Mon, Dec 2, 2019 at 6:54 PM Tyler Lubeck <ty...@tylerlubeck.com> wrote:
> Hi!
>
> I ran into an issue over the weekend where our automated system updated
> some of our brokers with a certificate signed by a new certificate
> authority but didn't update the truststore on the other brokers. I've
> dumped the stacktrace in a gist
> <https://gist.github.com/TylerLubeck/a734514d8a2e9936f35e223f90f21307> for
> context.
>
> From what I've been able to tell, this error only shows up in logs. It'd be
> useful to have this reported via other metrics as well so I can detect the
> problem sooner next time around.
>
> I *think* it'll be something like updating the SSLTransportLayer
> <
> https://github.com/apache/kafka/blob/be58580e14be93618f11e609389ff6bb16317702/clients/src/main/java/org/apache/kafka/common/network/SslTransportLayer.java#L50
> >
> to
> have a counter of number of failures and then registering itself as an
> mbean via CoreUtils
> <
> https://github.com/apache/kafka/blob/33d06082117d971cdcddd4f01392006b543f3c01/core/src/main/scala/kafka/utils/CoreUtils.scala#L140
> >.
> That said, I'm new to Java development and have only touched JMX from the
> 'oh neat, metrics' side of things.
>
> Is this worth pursuing? Is this roughly the right track? I'm more than
> happy to add this in but I'm not entirely sure how to start.
>
> Thanks,
> Tyler Lubeck
> (813) 469 - 1499
> www.TylerLubeck.com <http://www.tylerlubeck.com/>
> www.linkedin.com/in/tylerlubeck/
>