You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Christian <en...@gmail.com> on 2017/01/19 18:37:12 UTC
Kafka SASL_PLAINTEXT and authentication/authorization backend failure
I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster. We
implemented our own LoginModule and Server with the following caveat that I
am guessing I am doing something wrong.
The LoginModule's login method acquires a session id from an internal
security system and populates the subject with the relevant information. In
the server evaluateResponse we then validate that session. On success,
everything is great. However, when the evaulateResponse returns with a
failure (throws an exception), the producer client just hangs when sending
a message until the configured timeout occurs. Interestingly enough, we see
the evaulateResponse method is getting called about every second until the
the producer client finally times out.
We get this same behavior when using the PlainLoginModule provided with
Kafka after changing the password on the client side to something different
from the server side.
Is this expected behavior?
Thanks,
Christian
Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure
Posted by Christian <en...@gmail.com>.
I get this same behavior with Kafka 0.10.1.0 using PlainLoginModule and
simply making the password different from expected on the client. We also
get this behavior when creating our own Authorizer and always returning
false. I can tell a retry is happing because the brokers get called at
least every second. The problem is we can't find a way to get the client to
recognize this and re-initiate the original session from the client side.
So instead, we are forced to wait for timeout time.
On Thu, Jan 19, 2017 at 6:09 PM, Gerrit Jansen van Vuuren <
gerritjvv@gmail.com> wrote:
> Hi,
>
> I refer to the broker behaviour, for most part without SASL the brokers do
> respond but as soon as you put SASL into the mix it does hang whenever
> there is something that goes wrong, i.e the broker doesn't provide a
> response on the socket and the client waits reading a response that it
> never gets and timeout.
>
> e.g
>
> kafka 0.9 SASL protocol is [kerberos specific interchange] <normal kafka
> protocol interchange>
> kafka 0.10 is [len][kafka protocol][kerberos specific
> interchange] <normal kafka protocol interchange>
>
> the kerberos specific interchange parts is where things can hang (meaning
> the client or broker are waiting for information from either that never
> arrives), and if you write [len][kafka protocol] to a SASL configured 0.9
> kafka broker or vice versa the client timeout eventually not on retrying
> but on waiting on a response from the broker.
>
> I would file a bug but don't think this behaviour can be helped here.
>
> Regards,
> Gerrit
>
>
>
>
> On Fri, Jan 20, 2017 at 12:05 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Gerrit,
> >
> > I think it's important to distinguish broker and client behaviour. The
> > clients can hang because they keep retrying when they get certain errors.
> > When it comes to the broker, it should give you errors as a general rule.
> > If you are aware of certain scenarios where it should give an error and
> it
> > doesn't, then please file a bug with steps to reproduce.
> >
> > Ismael
> >
> > On Thu, Jan 19, 2017 at 6:48 PM, Gerrit Jansen van Vuuren <
> > gerritjvv@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I've added kerberos support for https://github.com/gerritjvv/
> kafka-fast
> > > and
> > > have seen that the kafka brokers do not send any response if the SASL
> > > authentication is not correct or accepted, thus causing the client to
> > hang
> > > while waiting for a response from kafka.
> > >
> > > Some things that might help to debug:
> > >
> > > - kafka 0.9's SASL auth is in-compatible with 0.10 and not using the
> > > correct version will cause the kafka client to hang.
> > > - use -Dsun.security.krb5.debug=true and
> > > -Djava.security.debug=gssloginconfig,configfile,
> > configparser,logincontext
> > > to see debug info about what's going on.
> > >
> > >
> > > Some reading material can be found at:
> > > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> > clj/Kerberos.md
> > >
> > > and if you want to see or need for testing a vagrant env with kerberos
> +
> > > kafka configured see
> > > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> > > clj/doc/vagrant.md
> > >
> > >
> > >
> > >
> > > On Thu, Jan 19, 2017 at 7:37 PM, Christian <en...@gmail.com> wrote:
> > >
> > > > I have successfully gotten SASL_PLAINTEXT configured on Kafka
> cluster.
> > We
> > > > implemented our own LoginModule and Server with the following caveat
> > > that I
> > > > am guessing I am doing something wrong.
> > > >
> > > > The LoginModule's login method acquires a session id from an internal
> > > > security system and populates the subject with the relevant
> > information.
> > > In
> > > > the server evaluateResponse we then validate that session. On
> success,
> > > > everything is great. However, when the evaulateResponse returns with
> a
> > > > failure (throws an exception), the producer client just hangs when
> > > sending
> > > > a message until the configured timeout occurs. Interestingly enough,
> we
> > > see
> > > > the evaulateResponse method is getting called about every second
> until
> > > the
> > > > the producer client finally times out.
> > > >
> > > > We get this same behavior when using the PlainLoginModule provided
> with
> > > > Kafka after changing the password on the client side to something
> > > different
> > > > from the server side.
> > > >
> > > > Is this expected behavior?
> > > >
> > > > Thanks,
> > > > Christian
> > > >
> > >
> >
>
Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure
Posted by Gerrit Jansen van Vuuren <ge...@gmail.com>.
Hi,
I refer to the broker behaviour, for most part without SASL the brokers do
respond but as soon as you put SASL into the mix it does hang whenever
there is something that goes wrong, i.e the broker doesn't provide a
response on the socket and the client waits reading a response that it
never gets and timeout.
e.g
kafka 0.9 SASL protocol is [kerberos specific interchange] <normal kafka
protocol interchange>
kafka 0.10 is [len][kafka protocol][kerberos specific
interchange] <normal kafka protocol interchange>
the kerberos specific interchange parts is where things can hang (meaning
the client or broker are waiting for information from either that never
arrives), and if you write [len][kafka protocol] to a SASL configured 0.9
kafka broker or vice versa the client timeout eventually not on retrying
but on waiting on a response from the broker.
I would file a bug but don't think this behaviour can be helped here.
Regards,
Gerrit
On Fri, Jan 20, 2017 at 12:05 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Gerrit,
>
> I think it's important to distinguish broker and client behaviour. The
> clients can hang because they keep retrying when they get certain errors.
> When it comes to the broker, it should give you errors as a general rule.
> If you are aware of certain scenarios where it should give an error and it
> doesn't, then please file a bug with steps to reproduce.
>
> Ismael
>
> On Thu, Jan 19, 2017 at 6:48 PM, Gerrit Jansen van Vuuren <
> gerritjvv@gmail.com> wrote:
>
> > Hi,
> >
> > I've added kerberos support for https://github.com/gerritjvv/kafka-fast
> > and
> > have seen that the kafka brokers do not send any response if the SASL
> > authentication is not correct or accepted, thus causing the client to
> hang
> > while waiting for a response from kafka.
> >
> > Some things that might help to debug:
> >
> > - kafka 0.9's SASL auth is in-compatible with 0.10 and not using the
> > correct version will cause the kafka client to hang.
> > - use -Dsun.security.krb5.debug=true and
> > -Djava.security.debug=gssloginconfig,configfile,
> configparser,logincontext
> > to see debug info about what's going on.
> >
> >
> > Some reading material can be found at:
> > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> clj/Kerberos.md
> >
> > and if you want to see or need for testing a vagrant env with kerberos +
> > kafka configured see
> > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> > clj/doc/vagrant.md
> >
> >
> >
> >
> > On Thu, Jan 19, 2017 at 7:37 PM, Christian <en...@gmail.com> wrote:
> >
> > > I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster.
> We
> > > implemented our own LoginModule and Server with the following caveat
> > that I
> > > am guessing I am doing something wrong.
> > >
> > > The LoginModule's login method acquires a session id from an internal
> > > security system and populates the subject with the relevant
> information.
> > In
> > > the server evaluateResponse we then validate that session. On success,
> > > everything is great. However, when the evaulateResponse returns with a
> > > failure (throws an exception), the producer client just hangs when
> > sending
> > > a message until the configured timeout occurs. Interestingly enough, we
> > see
> > > the evaulateResponse method is getting called about every second until
> > the
> > > the producer client finally times out.
> > >
> > > We get this same behavior when using the PlainLoginModule provided with
> > > Kafka after changing the password on the client side to something
> > different
> > > from the server side.
> > >
> > > Is this expected behavior?
> > >
> > > Thanks,
> > > Christian
> > >
> >
>
Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Gerrit,
I think it's important to distinguish broker and client behaviour. The
clients can hang because they keep retrying when they get certain errors.
When it comes to the broker, it should give you errors as a general rule.
If you are aware of certain scenarios where it should give an error and it
doesn't, then please file a bug with steps to reproduce.
Ismael
On Thu, Jan 19, 2017 at 6:48 PM, Gerrit Jansen van Vuuren <
gerritjvv@gmail.com> wrote:
> Hi,
>
> I've added kerberos support for https://github.com/gerritjvv/kafka-fast
> and
> have seen that the kafka brokers do not send any response if the SASL
> authentication is not correct or accepted, thus causing the client to hang
> while waiting for a response from kafka.
>
> Some things that might help to debug:
>
> - kafka 0.9's SASL auth is in-compatible with 0.10 and not using the
> correct version will cause the kafka client to hang.
> - use -Dsun.security.krb5.debug=true and
> -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext
> to see debug info about what's going on.
>
>
> Some reading material can be found at:
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/Kerberos.md
>
> and if you want to see or need for testing a vagrant env with kerberos +
> kafka configured see
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> clj/doc/vagrant.md
>
>
>
>
> On Thu, Jan 19, 2017 at 7:37 PM, Christian <en...@gmail.com> wrote:
>
> > I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster. We
> > implemented our own LoginModule and Server with the following caveat
> that I
> > am guessing I am doing something wrong.
> >
> > The LoginModule's login method acquires a session id from an internal
> > security system and populates the subject with the relevant information.
> In
> > the server evaluateResponse we then validate that session. On success,
> > everything is great. However, when the evaulateResponse returns with a
> > failure (throws an exception), the producer client just hangs when
> sending
> > a message until the configured timeout occurs. Interestingly enough, we
> see
> > the evaulateResponse method is getting called about every second until
> the
> > the producer client finally times out.
> >
> > We get this same behavior when using the PlainLoginModule provided with
> > Kafka after changing the password on the client side to something
> different
> > from the server side.
> >
> > Is this expected behavior?
> >
> > Thanks,
> > Christian
> >
>
Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure
Posted by Gerrit Jansen van Vuuren <ge...@gmail.com>.
If you mean authorization with kafka (not with kerberos) then yes, seems
that kafka stops responding when it doesn't get exactly what it expects :/
no errors just timeouts.
On Thu, Jan 19, 2017 at 8:59 PM, Christian <en...@gmail.com> wrote:
> Thanks for the response Gerrit! It seems like authorization has the same
> behavior. Have you experienced that as well?
>
> On Thu, Jan 19, 2017 at 11:48 AM, Gerrit Jansen van Vuuren <
> gerritjvv@gmail.com> wrote:
>
> > Hi,
> >
> > I've added kerberos support for https://github.com/gerritjvv/kafka-fast
> > and
> > have seen that the kafka brokers do not send any response if the SASL
> > authentication is not correct or accepted, thus causing the client to
> hang
> > while waiting for a response from kafka.
> >
> > Some things that might help to debug:
> >
> > - kafka 0.9's SASL auth is in-compatible with 0.10 and not using the
> > correct version will cause the kafka client to hang.
> > - use -Dsun.security.krb5.debug=true and
> > -Djava.security.debug=gssloginconfig,configfile,
> configparser,logincontext
> > to see debug info about what's going on.
> >
> >
> > Some reading material can be found at:
> > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> clj/Kerberos.md
> >
> > and if you want to see or need for testing a vagrant env with kerberos +
> > kafka configured see
> > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> > clj/doc/vagrant.md
> >
> >
> >
> >
> > On Thu, Jan 19, 2017 at 7:37 PM, Christian <en...@gmail.com> wrote:
> >
> > > I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster.
> We
> > > implemented our own LoginModule and Server with the following caveat
> > that I
> > > am guessing I am doing something wrong.
> > >
> > > The LoginModule's login method acquires a session id from an internal
> > > security system and populates the subject with the relevant
> information.
> > In
> > > the server evaluateResponse we then validate that session. On success,
> > > everything is great. However, when the evaulateResponse returns with a
> > > failure (throws an exception), the producer client just hangs when
> > sending
> > > a message until the configured timeout occurs. Interestingly enough, we
> > see
> > > the evaulateResponse method is getting called about every second until
> > the
> > > the producer client finally times out.
> > >
> > > We get this same behavior when using the PlainLoginModule provided with
> > > Kafka after changing the password on the client side to something
> > different
> > > from the server side.
> > >
> > > Is this expected behavior?
> > >
> > > Thanks,
> > > Christian
> > >
> >
>
Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure
Posted by Christian <en...@gmail.com>.
Thanks for the response Gerrit! It seems like authorization has the same
behavior. Have you experienced that as well?
On Thu, Jan 19, 2017 at 11:48 AM, Gerrit Jansen van Vuuren <
gerritjvv@gmail.com> wrote:
> Hi,
>
> I've added kerberos support for https://github.com/gerritjvv/kafka-fast
> and
> have seen that the kafka brokers do not send any response if the SASL
> authentication is not correct or accepted, thus causing the client to hang
> while waiting for a response from kafka.
>
> Some things that might help to debug:
>
> - kafka 0.9's SASL auth is in-compatible with 0.10 and not using the
> correct version will cause the kafka client to hang.
> - use -Dsun.security.krb5.debug=true and
> -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext
> to see debug info about what's going on.
>
>
> Some reading material can be found at:
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/Kerberos.md
>
> and if you want to see or need for testing a vagrant env with kerberos +
> kafka configured see
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-
> clj/doc/vagrant.md
>
>
>
>
> On Thu, Jan 19, 2017 at 7:37 PM, Christian <en...@gmail.com> wrote:
>
> > I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster. We
> > implemented our own LoginModule and Server with the following caveat
> that I
> > am guessing I am doing something wrong.
> >
> > The LoginModule's login method acquires a session id from an internal
> > security system and populates the subject with the relevant information.
> In
> > the server evaluateResponse we then validate that session. On success,
> > everything is great. However, when the evaulateResponse returns with a
> > failure (throws an exception), the producer client just hangs when
> sending
> > a message until the configured timeout occurs. Interestingly enough, we
> see
> > the evaulateResponse method is getting called about every second until
> the
> > the producer client finally times out.
> >
> > We get this same behavior when using the PlainLoginModule provided with
> > Kafka after changing the password on the client side to something
> different
> > from the server side.
> >
> > Is this expected behavior?
> >
> > Thanks,
> > Christian
> >
>
Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure
Posted by Gerrit Jansen van Vuuren <ge...@gmail.com>.
Hi,
I've added kerberos support for https://github.com/gerritjvv/kafka-fast and
have seen that the kafka brokers do not send any response if the SASL
authentication is not correct or accepted, thus causing the client to hang
while waiting for a response from kafka.
Some things that might help to debug:
- kafka 0.9's SASL auth is in-compatible with 0.10 and not using the
correct version will cause the kafka client to hang.
- use -Dsun.security.krb5.debug=true and
-Djava.security.debug=gssloginconfig,configfile,configparser,logincontext
to see debug info about what's going on.
Some reading material can be found at:
https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/Kerberos.md
and if you want to see or need for testing a vagrant env with kerberos +
kafka configured see
https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/doc/vagrant.md
On Thu, Jan 19, 2017 at 7:37 PM, Christian <en...@gmail.com> wrote:
> I have successfully gotten SASL_PLAINTEXT configured on Kafka cluster. We
> implemented our own LoginModule and Server with the following caveat that I
> am guessing I am doing something wrong.
>
> The LoginModule's login method acquires a session id from an internal
> security system and populates the subject with the relevant information. In
> the server evaluateResponse we then validate that session. On success,
> everything is great. However, when the evaulateResponse returns with a
> failure (throws an exception), the producer client just hangs when sending
> a message until the configured timeout occurs. Interestingly enough, we see
> the evaulateResponse method is getting called about every second until the
> the producer client finally times out.
>
> We get this same behavior when using the PlainLoginModule provided with
> Kafka after changing the password on the client side to something different
> from the server side.
>
> Is this expected behavior?
>
> Thanks,
> Christian
>