You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Manikumar <ma...@gmail.com> on 2017/08/01 06:08:19 UTC
Re: Adding new user to the broker dynamically
Server restart is required, only if you are using SASL/PLAIN mechanism.
Other mechanisms (Kerberos, Scram) restart is not required.
https://issues.apache.org/jira/browse/KAFKA-4292 will help us to write
custom handlers.
On Tue, Aug 1, 2017 at 4:26 AM, Alexei Levashov <
alexei.levashov@arrayent.com> wrote:
> Hello,
>
> Is there any dynamic approach to add user to the cluster for clients
> connecting to the running cluster.
> What I mean by that - can I avoid bouncing a broker if I have to add new
> user with say SASL authentication?
> When I add a new entry to kafka_server_jaas.conf it looks like it is
> required to bounce the broker for changes to take place.
>
> Thx,
> -AL
>
Re: Adding new user to the broker dynamically
Posted by Alexei Levashov <al...@arrayent.com>.
Hello Manikumar,
I set log level to debug and couldn't define appropriate filtering so logs
are a bit verbose.
I still can not interpret the failure.
Thx,
-AL
On Wed, Aug 2, 2017 at 12:01 AM, Manikumar <ma...@gmail.com>
wrote:
> looks like some config error. Can you upload initial logs for both the
> servers?
> One user is sufficient inter broker communication.
>
> On Wed, Aug 2, 2017 at 11:04 AM, Alexei Levashov <
> alexei.levashov@arrayent.com> wrote:
>
> > Hello Manikumar,
> >
> > I appreciate your advice , thank you.
> >
> > I tried to use SASL_PLAINTEXT with SCRAM enabled hoping that lack of SSL
> > will help debugging (will switch to SASL_SSL later).
> > I have 3 brokers running on one box with different ports
> > listeners = SASL_PLAINTEXT://<MY_IP>:9092
> > listeners = SASL_PLAINTEXT://<MY_IP>:9093
> > listeners = SASL_PLAINTEXT://<MY_IP>:9094
> >
> > 0. Changed broker.properties
> > listeners = SASL_PLAINTEXT://<MY_IP>:9093
> >
> > sasl.enabled.mechanisms = [SCRAM-SHA-256]
> > sasl.mechanism.inter.broker.protocol = SCRAM-SHA-256
> > security.inter.broker.protocol = SASL_PLAINTEXT
> >
> > 1.created admin user for the brokers
> > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
> > 'SCRAM-SHA-256=password=admin-secret,SCRAM-SHA-512=password=
> admin-secret'
> > --entity-type users --entity-name admin
> >
> > 2.created jaas.conf file in config dir :config/kafka_server_jaas.conf
> >
> > KafkaServer {
> > org.apache.kafka.common.security.plain.ScramLoginModule required
> > username="admin"
> > password="admin-secret"
> > user_admin="admin-secret"
> > user_alice="alice-secret";
> > };
> >
> > 3. Added export
> > KAFKA_OPTS="-Djava.security.auth.login.config=config/
> > kafka_server_jaas.conf"
> >
> > But I can start only one broker, the moment I start second broker I am
> > getting exceptions like these:
> >
> > [2017-08-02 04:30:36,733] DEBUG [Replica Manager on Broker 0]: Recording
> > follower broker 1 log read results:
> > ArrayBuffer((TNT_GRP_subgroup_getAttributeList_ACK-1,Fetch Data:
> > [FetchDataInfo(0 [0 : 0],[],false,None)], HW: [0], leaderLogStartOffset:
> > [0], leaderLogEndOffset: [0], followerLogStartOffset: [0], fetchTimeMs:
> > [1501648236733], readSize: [1048576], error: [NONE]))
> > (kafka.server.ReplicaManager)
> >
> > [2017-08-02 04:30:36,803] DEBUG Accepted connection from /<MY_IP>:58816
> on
> > /<MY_IP>:9093 and assigned it to processor 2, sendBufferSize
> > [actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
> > [102400|102400] (kafka.network.Acceptor)
> > [2017-08-02 04:30:36,803] DEBUG Processor 2 listening to new connection
> > from /<MY_IP>:58816 (kafka.network.Processor)
> > [2017-08-02 04:30:36,803] DEBUG Set SASL server state to
> HANDSHAKE_REQUEST
> > (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> > [2017-08-02 04:30:36,803] DEBUG Handle Kafka request METADATA
> > (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> > [2017-08-02 04:30:36,803] DEBUG Set SASL server state to FAILED
> > (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> > [2017-08-02 04:30:36,803] DEBUG Connection with /<MY_IP> disconnected
> > (org.apache.kafka.common.network.Selector)
> > java.io.IOException:
> > org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected
> Kafka
> > request of type METADATA during SASL handshake.
> > at
> > org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.
> > authenticate(SaslServerAuthenticator.java:247)
> > at
> > org.apache.kafka.common.network.KafkaChannel.prepare(
> KafkaChannel.java:76)
> > at
> > org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.
> > java:374)
> > at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
> > at kafka.network.Processor.poll(SocketServer.scala:499)
> > at kafka.network.Processor.run(SocketServer.scala:435)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.kafka.common.errors.IllegalSaslStateException:
> > Unexpected Kafka request of type METADATA during SASL handshake.
> > [2017-08-02 04:30:36,905] DEBUG Accepted connection from /<MY_IP>:58823
> on
> > /<MY_IP>:9093 and assigned it to processor 0, sendBufferSize
> > [actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
> > [102400|102400] (kafka.network.Acceptor)
> > [2017-08-02 04:30:36,905] DEBUG Processor 0 listening to new connection
> > from /<MY_IP>:58823 (kafka.network.Processor)
> > [2017-08-02 04:30:36,905] DEBUG Set SASL server state to
> HANDSHAKE_REQUEST
> > (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> > [2017-08-02 04:30:36,905] DEBUG Handle Kafka request METADATA
> > (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> > [2017-08-02 04:30:36,905] DEBUG Set SASL server state to FAILED
> > (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> > [2017-08-02 04:30:36,905] DEBUG Connection with /<MY_IP> disconnected
> > (org.apache.kafka.common.network.Selector)
> > java.io.IOException:
> > org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected
> Kafka
> > request of type METADATA during SASL handshake.
> >
> > Adding separate jaas.conf files for each broker with different users
> didn't
> > change anything.
> >
> > Question - should each broker use separate user for inter broker
> > communication? Or the reason for exceptions is broker set up on one IP?
> > Any hints would be highly appreciated.
> > Thx,
> > -AL
> >
> > On Mon, Jul 31, 2017 at 11:08 PM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > Server restart is required, only if you are using SASL/PLAIN
> mechanism.
> > > Other mechanisms (Kerberos, Scram) restart is not required.
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-4292 will help us to write
> > > custom handlers.
> > >
> > > On Tue, Aug 1, 2017 at 4:26 AM, Alexei Levashov <
> > > alexei.levashov@arrayent.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > Is there any dynamic approach to add user to the cluster for clients
> > > > connecting to the running cluster.
> > > > What I mean by that - can I avoid bouncing a broker if I have to add
> > new
> > > > user with say SASL authentication?
> > > > When I add a new entry to kafka_server_jaas.conf it looks like it is
> > > > required to bounce the broker for changes to take place.
> > > >
> > > > Thx,
> > > > -AL
> > > >
> > >
> >
>
Re: Adding new user to the broker dynamically
Posted by Manikumar <ma...@gmail.com>.
looks like some config error. Can you upload initial logs for both the
servers?
One user is sufficient inter broker communication.
On Wed, Aug 2, 2017 at 11:04 AM, Alexei Levashov <
alexei.levashov@arrayent.com> wrote:
> Hello Manikumar,
>
> I appreciate your advice , thank you.
>
> I tried to use SASL_PLAINTEXT with SCRAM enabled hoping that lack of SSL
> will help debugging (will switch to SASL_SSL later).
> I have 3 brokers running on one box with different ports
> listeners = SASL_PLAINTEXT://<MY_IP>:9092
> listeners = SASL_PLAINTEXT://<MY_IP>:9093
> listeners = SASL_PLAINTEXT://<MY_IP>:9094
>
> 0. Changed broker.properties
> listeners = SASL_PLAINTEXT://<MY_IP>:9093
>
> sasl.enabled.mechanisms = [SCRAM-SHA-256]
> sasl.mechanism.inter.broker.protocol = SCRAM-SHA-256
> security.inter.broker.protocol = SASL_PLAINTEXT
>
> 1.created admin user for the brokers
> bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
> 'SCRAM-SHA-256=password=admin-secret,SCRAM-SHA-512=password=admin-secret'
> --entity-type users --entity-name admin
>
> 2.created jaas.conf file in config dir :config/kafka_server_jaas.conf
>
> KafkaServer {
> org.apache.kafka.common.security.plain.ScramLoginModule required
> username="admin"
> password="admin-secret"
> user_admin="admin-secret"
> user_alice="alice-secret";
> };
>
> 3. Added export
> KAFKA_OPTS="-Djava.security.auth.login.config=config/
> kafka_server_jaas.conf"
>
> But I can start only one broker, the moment I start second broker I am
> getting exceptions like these:
>
> [2017-08-02 04:30:36,733] DEBUG [Replica Manager on Broker 0]: Recording
> follower broker 1 log read results:
> ArrayBuffer((TNT_GRP_subgroup_getAttributeList_ACK-1,Fetch Data:
> [FetchDataInfo(0 [0 : 0],[],false,None)], HW: [0], leaderLogStartOffset:
> [0], leaderLogEndOffset: [0], followerLogStartOffset: [0], fetchTimeMs:
> [1501648236733], readSize: [1048576], error: [NONE]))
> (kafka.server.ReplicaManager)
>
> [2017-08-02 04:30:36,803] DEBUG Accepted connection from /<MY_IP>:58816 on
> /<MY_IP>:9093 and assigned it to processor 2, sendBufferSize
> [actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
> [102400|102400] (kafka.network.Acceptor)
> [2017-08-02 04:30:36,803] DEBUG Processor 2 listening to new connection
> from /<MY_IP>:58816 (kafka.network.Processor)
> [2017-08-02 04:30:36,803] DEBUG Set SASL server state to HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> [2017-08-02 04:30:36,803] DEBUG Handle Kafka request METADATA
> (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> [2017-08-02 04:30:36,803] DEBUG Set SASL server state to FAILED
> (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> [2017-08-02 04:30:36,803] DEBUG Connection with /<MY_IP> disconnected
> (org.apache.kafka.common.network.Selector)
> java.io.IOException:
> org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected Kafka
> request of type METADATA during SASL handshake.
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.
> authenticate(SaslServerAuthenticator.java:247)
> at
> org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:76)
> at
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.
> java:374)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
> at kafka.network.Processor.poll(SocketServer.scala:499)
> at kafka.network.Processor.run(SocketServer.scala:435)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.IllegalSaslStateException:
> Unexpected Kafka request of type METADATA during SASL handshake.
> [2017-08-02 04:30:36,905] DEBUG Accepted connection from /<MY_IP>:58823 on
> /<MY_IP>:9093 and assigned it to processor 0, sendBufferSize
> [actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
> [102400|102400] (kafka.network.Acceptor)
> [2017-08-02 04:30:36,905] DEBUG Processor 0 listening to new connection
> from /<MY_IP>:58823 (kafka.network.Processor)
> [2017-08-02 04:30:36,905] DEBUG Set SASL server state to HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> [2017-08-02 04:30:36,905] DEBUG Handle Kafka request METADATA
> (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> [2017-08-02 04:30:36,905] DEBUG Set SASL server state to FAILED
> (org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
> [2017-08-02 04:30:36,905] DEBUG Connection with /<MY_IP> disconnected
> (org.apache.kafka.common.network.Selector)
> java.io.IOException:
> org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected Kafka
> request of type METADATA during SASL handshake.
>
> Adding separate jaas.conf files for each broker with different users didn't
> change anything.
>
> Question - should each broker use separate user for inter broker
> communication? Or the reason for exceptions is broker set up on one IP?
> Any hints would be highly appreciated.
> Thx,
> -AL
>
> On Mon, Jul 31, 2017 at 11:08 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > Server restart is required, only if you are using SASL/PLAIN mechanism.
> > Other mechanisms (Kerberos, Scram) restart is not required.
> >
> > https://issues.apache.org/jira/browse/KAFKA-4292 will help us to write
> > custom handlers.
> >
> > On Tue, Aug 1, 2017 at 4:26 AM, Alexei Levashov <
> > alexei.levashov@arrayent.com> wrote:
> >
> > > Hello,
> > >
> > > Is there any dynamic approach to add user to the cluster for clients
> > > connecting to the running cluster.
> > > What I mean by that - can I avoid bouncing a broker if I have to add
> new
> > > user with say SASL authentication?
> > > When I add a new entry to kafka_server_jaas.conf it looks like it is
> > > required to bounce the broker for changes to take place.
> > >
> > > Thx,
> > > -AL
> > >
> >
>
Re: Adding new user to the broker dynamically
Posted by Alexei Levashov <al...@arrayent.com>.
Hello Manikumar,
I appreciate your advice , thank you.
I tried to use SASL_PLAINTEXT with SCRAM enabled hoping that lack of SSL
will help debugging (will switch to SASL_SSL later).
I have 3 brokers running on one box with different ports
listeners = SASL_PLAINTEXT://<MY_IP>:9092
listeners = SASL_PLAINTEXT://<MY_IP>:9093
listeners = SASL_PLAINTEXT://<MY_IP>:9094
0. Changed broker.properties
listeners = SASL_PLAINTEXT://<MY_IP>:9093
sasl.enabled.mechanisms = [SCRAM-SHA-256]
sasl.mechanism.inter.broker.protocol = SCRAM-SHA-256
security.inter.broker.protocol = SASL_PLAINTEXT
1.created admin user for the brokers
bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
'SCRAM-SHA-256=password=admin-secret,SCRAM-SHA-512=password=admin-secret'
--entity-type users --entity-name admin
2.created jaas.conf file in config dir :config/kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.ScramLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret";
};
3. Added export
KAFKA_OPTS="-Djava.security.auth.login.config=config/kafka_server_jaas.conf"
But I can start only one broker, the moment I start second broker I am
getting exceptions like these:
[2017-08-02 04:30:36,733] DEBUG [Replica Manager on Broker 0]: Recording
follower broker 1 log read results:
ArrayBuffer((TNT_GRP_subgroup_getAttributeList_ACK-1,Fetch Data:
[FetchDataInfo(0 [0 : 0],[],false,None)], HW: [0], leaderLogStartOffset:
[0], leaderLogEndOffset: [0], followerLogStartOffset: [0], fetchTimeMs:
[1501648236733], readSize: [1048576], error: [NONE]))
(kafka.server.ReplicaManager)
[2017-08-02 04:30:36,803] DEBUG Accepted connection from /<MY_IP>:58816 on
/<MY_IP>:9093 and assigned it to processor 2, sendBufferSize
[actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
[102400|102400] (kafka.network.Acceptor)
[2017-08-02 04:30:36,803] DEBUG Processor 2 listening to new connection
from /<MY_IP>:58816 (kafka.network.Processor)
[2017-08-02 04:30:36,803] DEBUG Set SASL server state to HANDSHAKE_REQUEST
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-08-02 04:30:36,803] DEBUG Handle Kafka request METADATA
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-08-02 04:30:36,803] DEBUG Set SASL server state to FAILED
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-08-02 04:30:36,803] DEBUG Connection with /<MY_IP> disconnected
(org.apache.kafka.common.network.Selector)
java.io.IOException:
org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected Kafka
request of type METADATA during SASL handshake.
at
org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:247)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:76)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:374)
at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
at kafka.network.Processor.poll(SocketServer.scala:499)
at kafka.network.Processor.run(SocketServer.scala:435)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.IllegalSaslStateException:
Unexpected Kafka request of type METADATA during SASL handshake.
[2017-08-02 04:30:36,905] DEBUG Accepted connection from /<MY_IP>:58823 on
/<MY_IP>:9093 and assigned it to processor 0, sendBufferSize
[actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
[102400|102400] (kafka.network.Acceptor)
[2017-08-02 04:30:36,905] DEBUG Processor 0 listening to new connection
from /<MY_IP>:58823 (kafka.network.Processor)
[2017-08-02 04:30:36,905] DEBUG Set SASL server state to HANDSHAKE_REQUEST
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-08-02 04:30:36,905] DEBUG Handle Kafka request METADATA
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-08-02 04:30:36,905] DEBUG Set SASL server state to FAILED
(org.apache.kafka.common.security.authenticator.SaslServerAuthenticator)
[2017-08-02 04:30:36,905] DEBUG Connection with /<MY_IP> disconnected
(org.apache.kafka.common.network.Selector)
java.io.IOException:
org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected Kafka
request of type METADATA during SASL handshake.
Adding separate jaas.conf files for each broker with different users didn't
change anything.
Question - should each broker use separate user for inter broker
communication? Or the reason for exceptions is broker set up on one IP?
Any hints would be highly appreciated.
Thx,
-AL
On Mon, Jul 31, 2017 at 11:08 PM, Manikumar <ma...@gmail.com>
wrote:
> Server restart is required, only if you are using SASL/PLAIN mechanism.
> Other mechanisms (Kerberos, Scram) restart is not required.
>
> https://issues.apache.org/jira/browse/KAFKA-4292 will help us to write
> custom handlers.
>
> On Tue, Aug 1, 2017 at 4:26 AM, Alexei Levashov <
> alexei.levashov@arrayent.com> wrote:
>
> > Hello,
> >
> > Is there any dynamic approach to add user to the cluster for clients
> > connecting to the running cluster.
> > What I mean by that - can I avoid bouncing a broker if I have to add new
> > user with say SASL authentication?
> > When I add a new entry to kafka_server_jaas.conf it looks like it is
> > required to bounce the broker for changes to take place.
> >
> > Thx,
> > -AL
> >
>