You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Darshan <pu...@gmail.com> on 2018/04/03 21:45:21 UTC
Re: advertised.listeners
Hi Rajini
The above configuration that you mentioned a while back helped me sort the
issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL and
ACLs as well from one of your other posts.
I wanted to ask you if it is possible to run Kafka in a mixed security
mode? i.e. external producers who are on 172.x.x.x interface can use the
SSL to send/receive data from/to our Kafka brokers, but our internal
consumer can read/write on PLAINTEXT channel to Kafka.
Here is my server.properties, but my internal producer which does not have
any keystore and truststore is getting the following error:
*[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
(org.apache.kafka.clients.NetworkClient)*
*server.properties for our Kafka-1,2,3. They are identical except broker.id
<http://broker.id> and super.users properties.*
# ID and basic topic creation
broker.id=1
auto.create.topics.enable=true
delete.topic.enable=true
# LISTERN Settings
listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
172.21.190.176:9093
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
inter.broker.listener.name=INTERNAL
host.name=172.21.190.176
# Security Settings
ssl.keystore.location=keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=truststore.jks
ssl.truststore.password=password
ssl.keystore.type=JKS
ssl.truststore.type=JKS
security.protocol=SSL
ssl.client.auth=required
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=Kafka1
Can you please point out if anything needs to be modified ?
Many thanks.
--Darshan
On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <ra...@gmail.com>
wrote:
> If you want to use different interfaces with the same security protocol,
> you can specify listener names. You can then also configure different
> security properties for internal/external if you need.
>
> listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
>
> advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
>
> listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
>
> inter.broker.listener.name=INTERNAL
>
> On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com> wrote:
>
> > Hello Darshan
> >
> > Have you tried SSL://0.0.0.0:9093 ?
> >
> > Rajani had suggested something similar to me a week back while I was
> > trying to get a ACL based setup.
> >
> > Thanks.
> >
> > On Wed, May 31, 2017 at 8:58 AM, Darshan <pu...@gmail.com>
> > wrote:
> >
> >> Hi
> >>
> >> Our Kafka broker has two IPs on two different interfaces.
> >>
> >> eth0 has 172.x.x.x for external leg
> >> eth1 has 1.x.x.x for internal leg
> >>
> >>
> >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on 1.x.x.x
> >> subnet.
> >>
> >> If we use advertised.listeners=SSL://172.x.x.x:9093, then Producer can
> >> producer the message, but Consumer cannot receive the message.
> >>
> >> What value should we use for advertised.listeners so that Producer can
> >> write and Consumers can read ?
> >>
> >> Thanks.
> >>
> >
> >
> >
> > --
> > Raghav
> >
>
Re: advertised.listeners
Posted by Martin Gainty <mg...@hotmail.com>.
________________________________
From: Joe Hammerman <jh...@squarespace.com.INVALID>
Sent: Tuesday, April 3, 2018 10:40 PM
To: users@kafka.apache.org
Subject: Re: advertised.listeners
Hi all,
Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL?
MG>setup SSL on kafka
https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka
[https://avatars1.githubusercontent.com/u/524990?s=400&v=4]<https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka>
Using SSL with librdkafka · edenhill/librdkafka Wiki · GitHub<https://github.com/edenhill/librdkafka/wiki/Using-SSL-with-librdkafka>
github.com
The SSL support in librdkafka is completely configuration based, no new APIs are introduced, this means that any existing applications dynamically linked with librdkafka will get automatic SSL support by upgrading only the library. This page serves as a simple HOWTO guide. Apache Kafka brokers ...
MG> and then enabling SSL on listeners and advertised.listeners
MG>kafka PLAINTEXT and SSL server properties:
listeners=PLAINTEXT://:9093,SSL://:9092
advertised.listeners = PLAINTEXT://IP_ADDRESS:9093,SSL://IP_ADDRESS:9092
producerMB>security.protocol=PLAINTEXT
consumerMB>security.protocol=PLAINTEXT
MG>
What
should port and advertised listeners values in kafka-server.properties be
set in order to configure such an access profile? We want to so in order to
be able to perform healthchecking on the loopback device without having to
negotiate an SSL connection.
Thanks in advance for any assistance anyone can provide,
MG>PLAINTEXT for advertised.listeners is inherently dangerous since PLAINTEXT is allowed for MB but not for ZK
MG>Moreover SSL (not SASL-SSL) is only supported in ZK 3.5 which is alpha so CAVEAT EMPTOR
MG>if ZK is SSL enabled
MG>you will need a entity placed between MB and ZK to prevent ZK returning auth-fail on PLAINTEXT requests
MG>from MB
MG>"A current workaround while Kafka and ZooKeeper security features are not available is to have a secured Kafka cluster which coexists with PLAINTEXT clients such as the REST proxy."
MG>the suggested workaround for supporting PLAINTEXT and SSL
MG is to config
MG>network-segmentation (CIDR?)
http://www.itgeared.com/articles/1347-cidr-and-subnetting-tutorial/
CIDR and Subnetting | ITGeared.com<http://www.itgeared.com/articles/1347-cidr-and-subnetting-tutorial/>
www.itgeared.com
Classless Inter Domain Routing (CIDR) is a method for assigning IP addresses without using the standard IP address classes like Class A, Class B or Class C. CIDR is a newer addressing scheme for IP Networks which allows for a more efficient allocation of IP addresses than the older method which was by assigning organizations a class of IPs.
MG>if you dont want network segmentation you can config your Kafka ACLs explained here:
https://docs.confluent.io/3.0.0/kafka/platform-security.html
MG>HTH
Joseph Hammerman
On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mg...@hotmail.com> wrote:
> (from MessageBroker consumer)
>
> >tracert MessageProducer
>
> if zk server is found in tracert
>
> then yes you have a MB quorom
>
> FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
> https://stackoverflow.com/questions/46912937/is-it-
[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/46912937/is-it->
Is it possible to connect zookeeper and kafka via SASL , kafka broker and its clients via SSL without enabling plain text?<https://stackoverflow.com/questions/46912937/is-it->
stackoverflow.com
I am using Kafka Version 0.10.1. I connected Kafka brokers and its clients via SSL and its working fine.Now I have a query with some limitations. My limitations are No Plain text communications al...
> possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broker-and-its-cl
>
> but there is a ZK SASL SASL-Auth exception backdoor in
> zookeeper.properties when
> zookeeper.main_connection_despite_sasl_failure="yes"
>
>
> catch (SaslException e) {
> LOG.warn("Client failed to SASL authenticate: " + e);
> if ((System.getProperty("zookeeper.maintain_connection_despite_sasl_failure")
> != null)
> &&
> (System.getProperty("zookeeper.maintain_connection_
> despite_sasl_failure").equals("yes"))) {
> LOG.warn("Maintaining client connection despite SASL authentication
> failure.");
> } else {
> LOG.warn("Closing client connection due to SASL authentication
> failure.");
> cnxn.close();
> ..
> }
>
>
> does this conform to your findings?
>
> M
> ______________________________________________
>
>
>
> ________________________________
> From: Darshan <pu...@gmail.com>
> Sent: Tuesday, April 3, 2018 6:47 PM
> To: users@kafka.apache.org
> Subject: Re: advertised.listeners
>
> We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
> piece of info ? Thanks.
>
> On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com> wrote:
>
> > tracert MessageBrokerIP
> >
> > do you see ZK server in the trace?
> >
> > if yes then you are running kafka-cluster
> >
> > (ZK does not support mixed mode but there is a backdoor
> > zookeeper.properties config attribute that allows plaintext clients to
> > bypass sasl auth)
> >
> > ?
> >
> > Martin
> > ______________________________________________
> >
> >
> >
> > ________________________________
> > From: Darshan <pu...@gmail.com>
> > Sent: Tuesday, April 3, 2018 5:45 PM
> > To: rajinisivaram@gmail.com
> > Cc: users@kafka.apache.org
> > Subject: Re: advertised.listeners
> >
> > Hi Rajini
> >
> > The above configuration that you mentioned a while back helped me sort
> the
> > issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL and
> > ACLs as well from one of your other posts.
> >
> > I wanted to ask you if it is possible to run Kafka in a mixed security
> > mode? i.e. external producers who are on 172.x.x.x interface can use the
> > SSL to send/receive data from/to our Kafka brokers, but our internal
> > consumer can read/write on PLAINTEXT channel to Kafka.
> >
> > Here is my server.properties, but my internal producer which does not
> have
> > any keystore and truststore is getting the following error:
> > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
> > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
> > (org.apache.kafka.clients.NetworkClient)*
> >
> >
> > *server.properties for our Kafka-1,2,3. They are identical except
> > broker.id
> > <http://broker.id> and super.users properties.*
> broker.id - This website is for sale! - broker Resources and
> Information.<http://broker.id/>
> broker.id
> This website is for sale! broker.id is your first and best source for all
> of the information you’re looking for. From general topics to more of what
> you would expect to find here, broker.id has it all. We hope you find
> what you are searching for!
>
>
>
> > broker.id - This website is for sale! - broker Resources and
> > Information.<http://broker.id/>
> > broker.id
> > This website is for sale! broker.id is your first and best source for
> all
> > of the information you’re looking for. From general topics to more of
> what
> > you would expect to find here, broker.id has it all. We hope you find
> > what you are searching for!
> >
> >
> >
> >
> > # ID and basic topic creation
> > broker.id=1
> > auto.create.topics.enable=true
> > delete.topic.enable=true
> >
> > # LISTERN Settings
> > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
> > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
> > 172.21.190.176:9093
> > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
> > inter.broker.listener.name=INTERNAL
> > host.name=172.21.190.176
> >
> > # Security Settings
> > ssl.keystore.location=keystore.jks
> > ssl.keystore.password=password
> > ssl.key.password=password
> > ssl.truststore.location=truststore.jks
> > ssl.truststore.password=password
> > ssl.keystore.type=JKS
> > ssl.truststore.type=JKS
> > security.protocol=SSL
> > ssl.client.auth=required
> > allow.everyone.if.no.acl.found=false
> > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > super.users=User:CN=Kafka1
> >
> > Can you please point out if anything needs to be modified ?
> >
> > Many thanks.
> >
> > --Darshan
> >
> > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <
> rajinisivaram@gmail.com>
> > wrote:
> >
> > > If you want to use different interfaces with the same security
> protocol,
> > > you can specify listener names. You can then also configure different
> > > security properties for internal/external if you need.
> > >
> > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> > >
> > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> > >
> > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
> > >
> > > inter.broker.listener.name=INTERNAL
> > >
> > > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com> wrote:
> > >
> > > > Hello Darshan
> > > >
> > > > Have you tried SSL://0.0.0.0:9093 ?
> > > >
> > > > Rajani had suggested something similar to me a week back while I was
> > > > trying to get a ACL based setup.
> > > >
> > > > Thanks.
> > > >
> > > > On Wed, May 31, 2017 at 8:58 AM, Darshan <
> purandare.darshan@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> Our Kafka broker has two IPs on two different interfaces.
> > > >>
> > > >> eth0 has 172.x.x.x for external leg
> > > >> eth1 has 1.x.x.x for internal leg
> > > >>
> > > >>
> > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
> > 1.x.x.x
> > > >> subnet.
> > > >>
> > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then Producer
> > can
> > > >> producer the message, but Consumer cannot receive the message.
> > > >>
> > > >> What value should we use for advertised.listeners so that Producer
> can
> > > >> write and Consumers can read ?
> > > >>
> > > >> Thanks.
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Raghav
> > > >
> > >
> >
>
Re: advertised.listeners
Posted by Darshan <pu...@gmail.com>.
Thanks, Manikumar for pointing the typo. Upon setting the ACL rules are
told by you above, it worked fine. I was able to run secure (on external
interface) and insecure (on internal interface) mode.
As a reference to others in the forum, in addition to the server.properties
that I have posted above, these are the two things I did:
1. Changed super.users to
*super.users=User:CN=Kafka1;UserANONYMOUS2. Used the ACL rule as told by
Manikumar:bin/kafka-acls.sh --authorizer-properties
zookeeper.connect=localhost:2181 --add --allow-principal User:ANONYMOUS
--allow-host \* --operation Read --topic test*
Many thanks Manikumar.
Thanks.
On Wed, Apr 4, 2018 at 7:25 PM, Manikumar <ma...@gmail.com> wrote:
> Hi,
>
> User name is ANONYMOUS, not CN=ANONYMOUS. You can enable authorizer logs
> (kafka-authorizer.log) and check for any deny errors.
> supers.users can be configured with same value across all brokers.
>
> sh kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
> --add --allow-principal User:ANONYMOUS --allow-host \* --operation Read
> --topic test
>
> On Thu, Apr 5, 2018 at 2:39 AM, Darshan <pu...@gmail.com>
> wrote:
>
>> Hi Manikumar
>>
>> I pushed ACLs for User:ANONYMOUS and when I list them they are listed as
>> shown. Can you please suggest if server.properties needs a change ?
>>
>> *[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-acls.sh
>> --authorizer-properties zookeeper.connect=Kafka1-1:2181 --list --topic
>> topic05*
>> *Current ACLs for resource `Topic:topic05`:*
>> * User:CN=ANONYMOUS has Allow permission for operations: Describe
>> from hosts: **
>> * User:CN=Producer05 has Allow permission for operations: Write
>> from hosts: **
>> * User:CN=Producer05 has Allow permission for operations: Describe
>> from hosts: **
>> * User:CN=ANONYMOUS has Allow permission for operations: Write
>> from hosts: **
>>
>> When I start sending messages using kafka console client, I see the
>> following error. topic05 is created with 16 partitions, but no directory is
>> created in logs directory. Can someone please help ?
>>
>> *[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-console-producer.sh
>> --broker-list Kafka1:9092 --topic topic05*
>> *WorldHello*
>> *[2018-04-04 21:01:40,338] WARN Error while fetching metadata with
>> correlation id 1 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:40,437] WARN Error while fetching metadata with
>> correlation id 2 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:40,538] WARN Error while fetching metadata with
>> correlation id 3 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:40,640] WARN Error while fetching metadata with
>> correlation id 4 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:40,740] WARN Error while fetching metadata with
>> correlation id 5 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:40,841] WARN Error while fetching metadata with
>> correlation id 6 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:40,943] WARN Error while fetching metadata with
>> correlation id 7 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>> *[2018-04-04 21:01:41,045] WARN Error while fetching metadata with
>> correlation id 8 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
>> (org.apache.kafka.clients.NetworkClient)*
>>
>>
>>
>>
>>
>>
>>
>> *server.properties for our Kafka-1,2,3. They are identical
>> except broker.id <http://broker.id/> and super.users properties.# ID and
>> basic topic creationbroker.id
>> <http://broker.id/>=1auto.create.topics.enable=truedelete.topic.enable=true#
>> LISTERN Settingslisteners=INTERNAL://1.1.1.165:9092
>> <http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093
>> <http://172.21.190.176:9093/>advertised.listeners=INTERNAL://1.1.1.165:9092
>> <http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093
>> <http://172.21.190.176:9093/>listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSLinter.broker.listener.name
>> <http://inter.broker.listener.name/>=INTERNALhost.name
>> <http://host.name/>=172.21.190.176# Security
>> Settingsssl.keystore.location=keystore.jksssl.keystore.password=passwordssl.key.password=passwordssl.truststore.location=truststore.jksssl.truststore.password=passwordssl.keystore.type=JKSssl.truststore.type=JKSsecurity.protocol=SSLssl.client.auth=requiredallow.everyone.if.no.acl.found=falseauthorizer.class.name
>> <http://authorizer.class.name/>=kafka.se
>> <http://kafka.se>curity.auth.SimpleAclAuthorizersuper.users=User:CN=Kafka1*
>>
>>
>> On Tue, Apr 3, 2018 at 10:42 PM, Manikumar <ma...@gmail.com>
>> wrote:
>>
>>> @Darshan,
>>> For PLAINTEXT channels, principal will be "ANONYMOUS". You need to give
>>> produce/consume permissions
>>> to "User:ANONYMOUS"
>>>
>>>
>>> On Wed, Apr 4, 2018 at 8:10 AM, Joe Hammerman <
>>> jhammerman@squarespace.com.invalid> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL?
>>> What
>>> > should port and advertised listeners values in kafka-server.properties
>>> be
>>> > set in order to configure such an access profile? We want to so in
>>> order to
>>> > be able to perform healthchecking on the loopback device without
>>> having to
>>> > negotiate an SSL connection.
>>> >
>>> > Thanks in advance for any assistance anyone can provide,
>>> > Joseph Hammerman
>>> >
>>> > On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mg...@hotmail.com>
>>> wrote:
>>> >
>>> > > (from MessageBroker consumer)
>>> > >
>>> > > >tracert MessageProducer
>>> > >
>>> > > if zk server is found in tracert
>>> > >
>>> > > then yes you have a MB quorom
>>> > >
>>> > > FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
>>> > > https://stackoverflow.com/questions/46912937/is-it-
>>> > > possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broke
>>> r-and-its-cl
>>> > >
>>> > > but there is a ZK SASL SASL-Auth exception backdoor in
>>> > > zookeeper.properties when
>>> > > zookeeper.main_connection_despite_sasl_failure="yes"
>>> > >
>>> > >
>>> > > catch (SaslException e) {
>>> > > LOG.warn("Client failed to SASL authenticate: " + e);
>>> > > if ((System.getProperty("zookeeper.maintain_connection_
>>> > despite_sasl_failure")
>>> > > != null)
>>> > > &&
>>> > > (System.getProperty("zookeeper.maintain_connection_
>>> > > despite_sasl_failure").equals("yes"))) {
>>> > > LOG.warn("Maintaining client connection despite SASL
>>> authentication
>>> > > failure.");
>>> > > } else {
>>> > > LOG.warn("Closing client connection due to SASL authentication
>>> > > failure.");
>>> > > cnxn.close();
>>> > > ..
>>> > > }
>>> > >
>>> > >
>>> > > does this conform to your findings?
>>> > >
>>> > > M
>>> > > ______________________________________________
>>> > >
>>> > >
>>> > >
>>> > > ________________________________
>>> > > From: Darshan <pu...@gmail.com>
>>> > > Sent: Tuesday, April 3, 2018 6:47 PM
>>> > > To: users@kafka.apache.org
>>> > > Subject: Re: advertised.listeners
>>> > >
>>> > > We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from
>>> this
>>> > > piece of info ? Thanks.
>>> > >
>>> > > On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com>
>>> > wrote:
>>> > >
>>> > > > tracert MessageBrokerIP
>>> > > >
>>> > > > do you see ZK server in the trace?
>>> > > >
>>> > > > if yes then you are running kafka-cluster
>>> > > >
>>> > > > (ZK does not support mixed mode but there is a backdoor
>>> > > > zookeeper.properties config attribute that allows plaintext
>>> clients to
>>> > > > bypass sasl auth)
>>> > > >
>>> > > > ?
>>> > > >
>>> > > > Martin
>>> > > > ______________________________________________
>>> > > >
>>> > > >
>>> > > >
>>> > > > ________________________________
>>> > > > From: Darshan <pu...@gmail.com>
>>> > > > Sent: Tuesday, April 3, 2018 5:45 PM
>>> > > > To: rajinisivaram@gmail.com
>>> > > > Cc: users@kafka.apache.org
>>> > > > Subject: Re: advertised.listeners
>>> > > >
>>> > > > Hi Rajini
>>> > > >
>>> > > > The above configuration that you mentioned a while back helped me
>>> sort
>>> > > the
>>> > > > issue of listeners and I was also able to run Kafka 0.10.2.1 with
>>> SSL
>>> > and
>>> > > > ACLs as well from one of your other posts.
>>> > > >
>>> > > > I wanted to ask you if it is possible to run Kafka in a mixed
>>> security
>>> > > > mode? i.e. external producers who are on 172.x.x.x interface can
>>> use
>>> > the
>>> > > > SSL to send/receive data from/to our Kafka brokers, but our
>>> internal
>>> > > > consumer can read/write on PLAINTEXT channel to Kafka.
>>> > > >
>>> > > > Here is my server.properties, but my internal producer which does
>>> not
>>> > > have
>>> > > > any keystore and truststore is getting the following error:
>>> > > > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
>>> > > > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
>>> > > > (org.apache.kafka.clients.NetworkClient)*
>>> > > >
>>> > > >
>>> > > > *server.properties for our Kafka-1,2,3. They are identical except
>>> > > > broker.id
>>> > > > <http://broker.id> and super.users properties.*
>>> > > broker.id - This website is for sale! - broker Resources and
>>> > > Information.<http://broker.id/>
>>> > > broker.id
>>> > > This website is for sale! broker.id is your first and best source
>>> for
>>> > all
>>> > > of the information you’re looking for. From general topics to more of
>>> > what
>>> > > you would expect to find here, broker.id has it all. We hope you
>>> find
>>> > > what you are searching for!
>>> > >
>>> > >
>>> > >
>>> > > > broker.id - This website is for sale! - broker Resources
>>> and
>>> > > > Information.<http://broker.id/>
>>> > > > broker.id
>>> > > > This website is for sale! broker.id is your first and best source
>>> for
>>> > > all
>>> > > > of the information you’re looking for. From general topics to more
>>> of
>>> > > what
>>> > > > you would expect to find here, broker.id has it all. We hope you
>>> find
>>> > > > what you are searching for!
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > # ID and basic topic creation
>>> > > > broker.id=1
>>> > > > auto.create.topics.enable=true
>>> > > > delete.topic.enable=true
>>> > > >
>>> > > > # LISTERN Settings
>>> > > > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
>>> > > > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
>>> > > > 172.21.190.176:9093
>>> > > > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
>>> > > > inter.broker.listener.name=INTERNAL
>>> > > > host.name=172.21.190.176
>>> > > >
>>> > > > # Security Settings
>>> > > > ssl.keystore.location=keystore.jks
>>> > > > ssl.keystore.password=password
>>> > > > ssl.key.password=password
>>> > > > ssl.truststore.location=truststore.jks
>>> > > > ssl.truststore.password=password
>>> > > > ssl.keystore.type=JKS
>>> > > > ssl.truststore.type=JKS
>>> > > > security.protocol=SSL
>>> > > > ssl.client.auth=required
>>> > > > allow.everyone.if.no.acl.found=false
>>> > > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>> > > > super.users=User:CN=Kafka1
>>> > > >
>>> > > > Can you please point out if anything needs to be modified ?
>>> > > >
>>> > > > Many thanks.
>>> > > >
>>> > > > --Darshan
>>> > > >
>>> > > > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <
>>> > > rajinisivaram@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > If you want to use different interfaces with the same security
>>> > > protocol,
>>> > > > > you can specify listener names. You can then also configure
>>> different
>>> > > > > security properties for internal/external if you need.
>>> > > > >
>>> > > > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
>>> > > > >
>>> > > > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.
>>> > x.x.x:9093
>>> > > > >
>>> > > > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
>>> > > > >
>>> > > > > inter.broker.listener.name=INTERNAL
>>> > > > >
>>> > > > > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com>
>>> > wrote:
>>> > > > >
>>> > > > > > Hello Darshan
>>> > > > > >
>>> > > > > > Have you tried SSL://0.0.0.0:9093 ?
>>> > > > > >
>>> > > > > > Rajani had suggested something similar to me a week back while
>>> I
>>> > was
>>> > > > > > trying to get a ACL based setup.
>>> > > > > >
>>> > > > > > Thanks.
>>> > > > > >
>>> > > > > > On Wed, May 31, 2017 at 8:58 AM, Darshan <
>>> > > purandare.darshan@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > >> Hi
>>> > > > > >>
>>> > > > > >> Our Kafka broker has two IPs on two different interfaces.
>>> > > > > >>
>>> > > > > >> eth0 has 172.x.x.x for external leg
>>> > > > > >> eth1 has 1.x.x.x for internal leg
>>> > > > > >>
>>> > > > > >>
>>> > > > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is
>>> on
>>> > > > 1.x.x.x
>>> > > > > >> subnet.
>>> > > > > >>
>>> > > > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then
>>> > Producer
>>> > > > can
>>> > > > > >> producer the message, but Consumer cannot receive the message.
>>> > > > > >>
>>> > > > > >> What value should we use for advertised.listeners so that
>>> Producer
>>> > > can
>>> > > > > >> write and Consumers can read ?
>>> > > > > >>
>>> > > > > >> Thanks.
>>> > > > > >>
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Raghav
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: advertised.listeners
Posted by Manikumar <ma...@gmail.com>.
Hi,
User name is ANONYMOUS, not CN=ANONYMOUS. You can enable authorizer logs
(kafka-authorizer.log) and check for any deny errors.
supers.users can be configured with same value across all brokers.
sh kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
--add --allow-principal User:ANONYMOUS --allow-host \* --operation Read
--topic test
On Thu, Apr 5, 2018 at 2:39 AM, Darshan <pu...@gmail.com> wrote:
> Hi Manikumar
>
> I pushed ACLs for User:ANONYMOUS and when I list them they are listed as
> shown. Can you please suggest if server.properties needs a change ?
>
> *[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-acls.sh
> --authorizer-properties zookeeper.connect=Kafka1-1:2181 --list --topic
> topic05*
> *Current ACLs for resource `Topic:topic05`:*
> * User:CN=ANONYMOUS has Allow permission for operations: Describe
> from hosts: **
> * User:CN=Producer05 has Allow permission for operations: Write
> from hosts: **
> * User:CN=Producer05 has Allow permission for operations: Describe
> from hosts: **
> * User:CN=ANONYMOUS has Allow permission for operations: Write from
> hosts: **
>
> When I start sending messages using kafka console client, I see the
> following error. topic05 is created with 16 partitions, but no directory is
> created in logs directory. Can someone please help ?
>
> *[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-console-producer.sh
> --broker-list Kafka1:9092 --topic topic05*
> *WorldHello*
> *[2018-04-04 21:01:40,338] WARN Error while fetching metadata with
> correlation id 1 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:40,437] WARN Error while fetching metadata with
> correlation id 2 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:40,538] WARN Error while fetching metadata with
> correlation id 3 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:40,640] WARN Error while fetching metadata with
> correlation id 4 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:40,740] WARN Error while fetching metadata with
> correlation id 5 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:40,841] WARN Error while fetching metadata with
> correlation id 6 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:40,943] WARN Error while fetching metadata with
> correlation id 7 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
> *[2018-04-04 21:01:41,045] WARN Error while fetching metadata with
> correlation id 8 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
> (org.apache.kafka.clients.NetworkClient)*
>
>
>
>
>
>
>
> *server.properties for our Kafka-1,2,3. They are identical
> except broker.id <http://broker.id/> and super.users properties.# ID and
> basic topic creationbroker.id
> <http://broker.id/>=1auto.create.topics.enable=truedelete.topic.enable=true#
> LISTERN Settingslisteners=INTERNAL://1.1.1.165:9092
> <http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093
> <http://172.21.190.176:9093/>advertised.listeners=INTERNAL://1.1.1.165:9092
> <http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093
> <http://172.21.190.176:9093/>listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSLinter.broker.listener.name
> <http://inter.broker.listener.name/>=INTERNALhost.name
> <http://host.name/>=172.21.190.176# Security
> Settingsssl.keystore.location=keystore.jksssl.keystore.password=passwordssl.key.password=passwordssl.truststore.location=truststore.jksssl.truststore.password=passwordssl.keystore.type=JKSssl.truststore.type=JKSsecurity.protocol=SSLssl.client.auth=requiredallow.everyone.if.no.acl.found=falseauthorizer.class.name
> <http://authorizer.class.name/>=kafka.se
> <http://kafka.se>curity.auth.SimpleAclAuthorizersuper.users=User:CN=Kafka1*
>
>
> On Tue, Apr 3, 2018 at 10:42 PM, Manikumar <ma...@gmail.com>
> wrote:
>
>> @Darshan,
>> For PLAINTEXT channels, principal will be "ANONYMOUS". You need to give
>> produce/consume permissions
>> to "User:ANONYMOUS"
>>
>>
>> On Wed, Apr 4, 2018 at 8:10 AM, Joe Hammerman <
>> jhammerman@squarespace.com.invalid> wrote:
>>
>> > Hi all,
>> >
>> > Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL?
>> What
>> > should port and advertised listeners values in kafka-server.properties
>> be
>> > set in order to configure such an access profile? We want to so in
>> order to
>> > be able to perform healthchecking on the loopback device without having
>> to
>> > negotiate an SSL connection.
>> >
>> > Thanks in advance for any assistance anyone can provide,
>> > Joseph Hammerman
>> >
>> > On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mg...@hotmail.com>
>> wrote:
>> >
>> > > (from MessageBroker consumer)
>> > >
>> > > >tracert MessageProducer
>> > >
>> > > if zk server is found in tracert
>> > >
>> > > then yes you have a MB quorom
>> > >
>> > > FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
>> > > https://stackoverflow.com/questions/46912937/is-it-
>> > > possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broke
>> r-and-its-cl
>> > >
>> > > but there is a ZK SASL SASL-Auth exception backdoor in
>> > > zookeeper.properties when
>> > > zookeeper.main_connection_despite_sasl_failure="yes"
>> > >
>> > >
>> > > catch (SaslException e) {
>> > > LOG.warn("Client failed to SASL authenticate: " + e);
>> > > if ((System.getProperty("zookeeper.maintain_connection_
>> > despite_sasl_failure")
>> > > != null)
>> > > &&
>> > > (System.getProperty("zookeeper.maintain_connection_
>> > > despite_sasl_failure").equals("yes"))) {
>> > > LOG.warn("Maintaining client connection despite SASL
>> authentication
>> > > failure.");
>> > > } else {
>> > > LOG.warn("Closing client connection due to SASL authentication
>> > > failure.");
>> > > cnxn.close();
>> > > ..
>> > > }
>> > >
>> > >
>> > > does this conform to your findings?
>> > >
>> > > M
>> > > ______________________________________________
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Darshan <pu...@gmail.com>
>> > > Sent: Tuesday, April 3, 2018 6:47 PM
>> > > To: users@kafka.apache.org
>> > > Subject: Re: advertised.listeners
>> > >
>> > > We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
>> > > piece of info ? Thanks.
>> > >
>> > > On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com>
>> > wrote:
>> > >
>> > > > tracert MessageBrokerIP
>> > > >
>> > > > do you see ZK server in the trace?
>> > > >
>> > > > if yes then you are running kafka-cluster
>> > > >
>> > > > (ZK does not support mixed mode but there is a backdoor
>> > > > zookeeper.properties config attribute that allows plaintext clients
>> to
>> > > > bypass sasl auth)
>> > > >
>> > > > ?
>> > > >
>> > > > Martin
>> > > > ______________________________________________
>> > > >
>> > > >
>> > > >
>> > > > ________________________________
>> > > > From: Darshan <pu...@gmail.com>
>> > > > Sent: Tuesday, April 3, 2018 5:45 PM
>> > > > To: rajinisivaram@gmail.com
>> > > > Cc: users@kafka.apache.org
>> > > > Subject: Re: advertised.listeners
>> > > >
>> > > > Hi Rajini
>> > > >
>> > > > The above configuration that you mentioned a while back helped me
>> sort
>> > > the
>> > > > issue of listeners and I was also able to run Kafka 0.10.2.1 with
>> SSL
>> > and
>> > > > ACLs as well from one of your other posts.
>> > > >
>> > > > I wanted to ask you if it is possible to run Kafka in a mixed
>> security
>> > > > mode? i.e. external producers who are on 172.x.x.x interface can use
>> > the
>> > > > SSL to send/receive data from/to our Kafka brokers, but our internal
>> > > > consumer can read/write on PLAINTEXT channel to Kafka.
>> > > >
>> > > > Here is my server.properties, but my internal producer which does
>> not
>> > > have
>> > > > any keystore and truststore is getting the following error:
>> > > > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
>> > > > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
>> > > > (org.apache.kafka.clients.NetworkClient)*
>> > > >
>> > > >
>> > > > *server.properties for our Kafka-1,2,3. They are identical except
>> > > > broker.id
>> > > > <http://broker.id> and super.users properties.*
>> > > broker.id - This website is for sale! - broker Resources and
>> > > Information.<http://broker.id/>
>> > > broker.id
>> > > This website is for sale! broker.id is your first and best source for
>> > all
>> > > of the information you’re looking for. From general topics to more of
>> > what
>> > > you would expect to find here, broker.id has it all. We hope you find
>> > > what you are searching for!
>> > >
>> > >
>> > >
>> > > > broker.id - This website is for sale! - broker Resources
>> and
>> > > > Information.<http://broker.id/>
>> > > > broker.id
>> > > > This website is for sale! broker.id is your first and best source
>> for
>> > > all
>> > > > of the information you’re looking for. From general topics to more
>> of
>> > > what
>> > > > you would expect to find here, broker.id has it all. We hope you
>> find
>> > > > what you are searching for!
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > # ID and basic topic creation
>> > > > broker.id=1
>> > > > auto.create.topics.enable=true
>> > > > delete.topic.enable=true
>> > > >
>> > > > # LISTERN Settings
>> > > > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
>> > > > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
>> > > > 172.21.190.176:9093
>> > > > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
>> > > > inter.broker.listener.name=INTERNAL
>> > > > host.name=172.21.190.176
>> > > >
>> > > > # Security Settings
>> > > > ssl.keystore.location=keystore.jks
>> > > > ssl.keystore.password=password
>> > > > ssl.key.password=password
>> > > > ssl.truststore.location=truststore.jks
>> > > > ssl.truststore.password=password
>> > > > ssl.keystore.type=JKS
>> > > > ssl.truststore.type=JKS
>> > > > security.protocol=SSL
>> > > > ssl.client.auth=required
>> > > > allow.everyone.if.no.acl.found=false
>> > > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>> > > > super.users=User:CN=Kafka1
>> > > >
>> > > > Can you please point out if anything needs to be modified ?
>> > > >
>> > > > Many thanks.
>> > > >
>> > > > --Darshan
>> > > >
>> > > > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <
>> > > rajinisivaram@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > If you want to use different interfaces with the same security
>> > > protocol,
>> > > > > you can specify listener names. You can then also configure
>> different
>> > > > > security properties for internal/external if you need.
>> > > > >
>> > > > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
>> > > > >
>> > > > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.
>> > x.x.x:9093
>> > > > >
>> > > > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
>> > > > >
>> > > > > inter.broker.listener.name=INTERNAL
>> > > > >
>> > > > > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > Hello Darshan
>> > > > > >
>> > > > > > Have you tried SSL://0.0.0.0:9093 ?
>> > > > > >
>> > > > > > Rajani had suggested something similar to me a week back while I
>> > was
>> > > > > > trying to get a ACL based setup.
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > On Wed, May 31, 2017 at 8:58 AM, Darshan <
>> > > purandare.darshan@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > >> Hi
>> > > > > >>
>> > > > > >> Our Kafka broker has two IPs on two different interfaces.
>> > > > > >>
>> > > > > >> eth0 has 172.x.x.x for external leg
>> > > > > >> eth1 has 1.x.x.x for internal leg
>> > > > > >>
>> > > > > >>
>> > > > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
>> > > > 1.x.x.x
>> > > > > >> subnet.
>> > > > > >>
>> > > > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then
>> > Producer
>> > > > can
>> > > > > >> producer the message, but Consumer cannot receive the message.
>> > > > > >>
>> > > > > >> What value should we use for advertised.listeners so that
>> Producer
>> > > can
>> > > > > >> write and Consumers can read ?
>> > > > > >>
>> > > > > >> Thanks.
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Raghav
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: advertised.listeners
Posted by Darshan <pu...@gmail.com>.
Hi Manikumar
I pushed ACLs for User:ANONYMOUS and when I list them they are listed as
shown. Can you please suggest if server.properties needs a change ?
*[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-acls.sh
--authorizer-properties zookeeper.connect=Kafka1-1:2181 --list --topic
topic05*
*Current ACLs for resource `Topic:topic05`:*
* User:CN=ANONYMOUS has Allow permission for operations: Describe
from hosts: **
* User:CN=Producer05 has Allow permission for operations: Write from
hosts: **
* User:CN=Producer05 has Allow permission for operations: Describe
from hosts: **
* User:CN=ANONYMOUS has Allow permission for operations: Write from
hosts: **
When I start sending messages using kafka console client, I see the
following error. topic05 is created with 16 partitions, but no directory is
created in logs directory. Can someone please help ?
*[alpha: user@Kafka1 kafka_2.12-0.10.2.1]$ bin/kafka-console-producer.sh
--broker-list Kafka1:9092 --topic topic05*
*WorldHello*
*[2018-04-04 21:01:40,338] WARN Error while fetching metadata with
correlation id 1 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:40,437] WARN Error while fetching metadata with
correlation id 2 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:40,538] WARN Error while fetching metadata with
correlation id 3 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:40,640] WARN Error while fetching metadata with
correlation id 4 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:40,740] WARN Error while fetching metadata with
correlation id 5 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:40,841] WARN Error while fetching metadata with
correlation id 6 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:40,943] WARN Error while fetching metadata with
correlation id 7 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*[2018-04-04 21:01:41,045] WARN Error while fetching metadata with
correlation id 8 : {topic05=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)*
*server.properties for our Kafka-1,2,3. They are identical except broker.id
<http://broker.id/> and super.users properties.# ID and basic topic
creationbroker.id
<http://broker.id/>=1auto.create.topics.enable=truedelete.topic.enable=true#
LISTERN Settingslisteners=INTERNAL://1.1.1.165:9092
<http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093
<http://172.21.190.176:9093/>advertised.listeners=INTERNAL://1.1.1.165:9092
<http://1.1.1.165:9092/>,EXTERNAL://172.21.190.176:9093
<http://172.21.190.176:9093/>listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSLinter.broker.listener.name
<http://inter.broker.listener.name/>=INTERNALhost.name
<http://host.name/>=172.21.190.176# Security
Settingsssl.keystore.location=keystore.jksssl.keystore.password=passwordssl.key.password=passwordssl.truststore.location=truststore.jksssl.truststore.password=passwordssl.keystore.type=JKSssl.truststore.type=JKSsecurity.protocol=SSLssl.client.auth=requiredallow.everyone.if.no.acl.found=falseauthorizer.class.name
<http://authorizer.class.name/>=kafka.security.auth.SimpleAclAuthorizersuper.users=User:CN=Kafka1*
On Tue, Apr 3, 2018 at 10:42 PM, Manikumar <ma...@gmail.com>
wrote:
> @Darshan,
> For PLAINTEXT channels, principal will be "ANONYMOUS". You need to give
> produce/consume permissions
> to "User:ANONYMOUS"
>
>
> On Wed, Apr 4, 2018 at 8:10 AM, Joe Hammerman <
> jhammerman@squarespace.com.invalid> wrote:
>
> > Hi all,
> >
> > Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL?
> What
> > should port and advertised listeners values in kafka-server.properties be
> > set in order to configure such an access profile? We want to so in order
> to
> > be able to perform healthchecking on the loopback device without having
> to
> > negotiate an SSL connection.
> >
> > Thanks in advance for any assistance anyone can provide,
> > Joseph Hammerman
> >
> > On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mg...@hotmail.com>
> wrote:
> >
> > > (from MessageBroker consumer)
> > >
> > > >tracert MessageProducer
> > >
> > > if zk server is found in tracert
> > >
> > > then yes you have a MB quorom
> > >
> > > FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
> > > https://stackoverflow.com/questions/46912937/is-it-
> > > possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-
> broker-and-its-cl
> > >
> > > but there is a ZK SASL SASL-Auth exception backdoor in
> > > zookeeper.properties when
> > > zookeeper.main_connection_despite_sasl_failure="yes"
> > >
> > >
> > > catch (SaslException e) {
> > > LOG.warn("Client failed to SASL authenticate: " + e);
> > > if ((System.getProperty("zookeeper.maintain_connection_
> > despite_sasl_failure")
> > > != null)
> > > &&
> > > (System.getProperty("zookeeper.maintain_connection_
> > > despite_sasl_failure").equals("yes"))) {
> > > LOG.warn("Maintaining client connection despite SASL
> authentication
> > > failure.");
> > > } else {
> > > LOG.warn("Closing client connection due to SASL authentication
> > > failure.");
> > > cnxn.close();
> > > ..
> > > }
> > >
> > >
> > > does this conform to your findings?
> > >
> > > M
> > > ______________________________________________
> > >
> > >
> > >
> > > ________________________________
> > > From: Darshan <pu...@gmail.com>
> > > Sent: Tuesday, April 3, 2018 6:47 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: advertised.listeners
> > >
> > > We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
> > > piece of info ? Thanks.
> > >
> > > On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com>
> > wrote:
> > >
> > > > tracert MessageBrokerIP
> > > >
> > > > do you see ZK server in the trace?
> > > >
> > > > if yes then you are running kafka-cluster
> > > >
> > > > (ZK does not support mixed mode but there is a backdoor
> > > > zookeeper.properties config attribute that allows plaintext clients
> to
> > > > bypass sasl auth)
> > > >
> > > > ?
> > > >
> > > > Martin
> > > > ______________________________________________
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > From: Darshan <pu...@gmail.com>
> > > > Sent: Tuesday, April 3, 2018 5:45 PM
> > > > To: rajinisivaram@gmail.com
> > > > Cc: users@kafka.apache.org
> > > > Subject: Re: advertised.listeners
> > > >
> > > > Hi Rajini
> > > >
> > > > The above configuration that you mentioned a while back helped me
> sort
> > > the
> > > > issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL
> > and
> > > > ACLs as well from one of your other posts.
> > > >
> > > > I wanted to ask you if it is possible to run Kafka in a mixed
> security
> > > > mode? i.e. external producers who are on 172.x.x.x interface can use
> > the
> > > > SSL to send/receive data from/to our Kafka brokers, but our internal
> > > > consumer can read/write on PLAINTEXT channel to Kafka.
> > > >
> > > > Here is my server.properties, but my internal producer which does not
> > > have
> > > > any keystore and truststore is getting the following error:
> > > > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
> > > > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
> > > > (org.apache.kafka.clients.NetworkClient)*
> > > >
> > > >
> > > > *server.properties for our Kafka-1,2,3. They are identical except
> > > > broker.id
> > > > <http://broker.id> and super.users properties.*
> > > broker.id - This website is for sale! - broker Resources and
> > > Information.<http://broker.id/>
> > > broker.id
> > > This website is for sale! broker.id is your first and best source for
> > all
> > > of the information you’re looking for. From general topics to more of
> > what
> > > you would expect to find here, broker.id has it all. We hope you find
> > > what you are searching for!
> > >
> > >
> > >
> > > > broker.id - This website is for sale! - broker Resources and
> > > > Information.<http://broker.id/>
> > > > broker.id
> > > > This website is for sale! broker.id is your first and best source
> for
> > > all
> > > > of the information you’re looking for. From general topics to more of
> > > what
> > > > you would expect to find here, broker.id has it all. We hope you
> find
> > > > what you are searching for!
> > > >
> > > >
> > > >
> > > >
> > > > # ID and basic topic creation
> > > > broker.id=1
> > > > auto.create.topics.enable=true
> > > > delete.topic.enable=true
> > > >
> > > > # LISTERN Settings
> > > > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
> > > > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
> > > > 172.21.190.176:9093
> > > > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
> > > > inter.broker.listener.name=INTERNAL
> > > > host.name=172.21.190.176
> > > >
> > > > # Security Settings
> > > > ssl.keystore.location=keystore.jks
> > > > ssl.keystore.password=password
> > > > ssl.key.password=password
> > > > ssl.truststore.location=truststore.jks
> > > > ssl.truststore.password=password
> > > > ssl.keystore.type=JKS
> > > > ssl.truststore.type=JKS
> > > > security.protocol=SSL
> > > > ssl.client.auth=required
> > > > allow.everyone.if.no.acl.found=false
> > > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > > > super.users=User:CN=Kafka1
> > > >
> > > > Can you please point out if anything needs to be modified ?
> > > >
> > > > Many thanks.
> > > >
> > > > --Darshan
> > > >
> > > > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <
> > > rajinisivaram@gmail.com>
> > > > wrote:
> > > >
> > > > > If you want to use different interfaces with the same security
> > > protocol,
> > > > > you can specify listener names. You can then also configure
> different
> > > > > security properties for internal/external if you need.
> > > > >
> > > > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> > > > >
> > > > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.
> > x.x.x:9093
> > > > >
> > > > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
> > > > >
> > > > > inter.broker.listener.name=INTERNAL
> > > > >
> > > > > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hello Darshan
> > > > > >
> > > > > > Have you tried SSL://0.0.0.0:9093 ?
> > > > > >
> > > > > > Rajani had suggested something similar to me a week back while I
> > was
> > > > > > trying to get a ACL based setup.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > On Wed, May 31, 2017 at 8:58 AM, Darshan <
> > > purandare.darshan@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi
> > > > > >>
> > > > > >> Our Kafka broker has two IPs on two different interfaces.
> > > > > >>
> > > > > >> eth0 has 172.x.x.x for external leg
> > > > > >> eth1 has 1.x.x.x for internal leg
> > > > > >>
> > > > > >>
> > > > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
> > > > 1.x.x.x
> > > > > >> subnet.
> > > > > >>
> > > > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then
> > Producer
> > > > can
> > > > > >> producer the message, but Consumer cannot receive the message.
> > > > > >>
> > > > > >> What value should we use for advertised.listeners so that
> Producer
> > > can
> > > > > >> write and Consumers can read ?
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Raghav
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: advertised.listeners
Posted by Manikumar <ma...@gmail.com>.
@Darshan,
For PLAINTEXT channels, principal will be "ANONYMOUS". You need to give
produce/consume permissions
to "User:ANONYMOUS"
On Wed, Apr 4, 2018 at 8:10 AM, Joe Hammerman <
jhammerman@squarespace.com.invalid> wrote:
> Hi all,
>
> Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL? What
> should port and advertised listeners values in kafka-server.properties be
> set in order to configure such an access profile? We want to so in order to
> be able to perform healthchecking on the loopback device without having to
> negotiate an SSL connection.
>
> Thanks in advance for any assistance anyone can provide,
> Joseph Hammerman
>
> On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mg...@hotmail.com> wrote:
>
> > (from MessageBroker consumer)
> >
> > >tracert MessageProducer
> >
> > if zk server is found in tracert
> >
> > then yes you have a MB quorom
> >
> > FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
> > https://stackoverflow.com/questions/46912937/is-it-
> > possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broker-and-its-cl
> >
> > but there is a ZK SASL SASL-Auth exception backdoor in
> > zookeeper.properties when
> > zookeeper.main_connection_despite_sasl_failure="yes"
> >
> >
> > catch (SaslException e) {
> > LOG.warn("Client failed to SASL authenticate: " + e);
> > if ((System.getProperty("zookeeper.maintain_connection_
> despite_sasl_failure")
> > != null)
> > &&
> > (System.getProperty("zookeeper.maintain_connection_
> > despite_sasl_failure").equals("yes"))) {
> > LOG.warn("Maintaining client connection despite SASL authentication
> > failure.");
> > } else {
> > LOG.warn("Closing client connection due to SASL authentication
> > failure.");
> > cnxn.close();
> > ..
> > }
> >
> >
> > does this conform to your findings?
> >
> > M
> > ______________________________________________
> >
> >
> >
> > ________________________________
> > From: Darshan <pu...@gmail.com>
> > Sent: Tuesday, April 3, 2018 6:47 PM
> > To: users@kafka.apache.org
> > Subject: Re: advertised.listeners
> >
> > We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
> > piece of info ? Thanks.
> >
> > On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com>
> wrote:
> >
> > > tracert MessageBrokerIP
> > >
> > > do you see ZK server in the trace?
> > >
> > > if yes then you are running kafka-cluster
> > >
> > > (ZK does not support mixed mode but there is a backdoor
> > > zookeeper.properties config attribute that allows plaintext clients to
> > > bypass sasl auth)
> > >
> > > ?
> > >
> > > Martin
> > > ______________________________________________
> > >
> > >
> > >
> > > ________________________________
> > > From: Darshan <pu...@gmail.com>
> > > Sent: Tuesday, April 3, 2018 5:45 PM
> > > To: rajinisivaram@gmail.com
> > > Cc: users@kafka.apache.org
> > > Subject: Re: advertised.listeners
> > >
> > > Hi Rajini
> > >
> > > The above configuration that you mentioned a while back helped me sort
> > the
> > > issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL
> and
> > > ACLs as well from one of your other posts.
> > >
> > > I wanted to ask you if it is possible to run Kafka in a mixed security
> > > mode? i.e. external producers who are on 172.x.x.x interface can use
> the
> > > SSL to send/receive data from/to our Kafka brokers, but our internal
> > > consumer can read/write on PLAINTEXT channel to Kafka.
> > >
> > > Here is my server.properties, but my internal producer which does not
> > have
> > > any keystore and truststore is getting the following error:
> > > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
> > > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
> > > (org.apache.kafka.clients.NetworkClient)*
> > >
> > >
> > > *server.properties for our Kafka-1,2,3. They are identical except
> > > broker.id
> > > <http://broker.id> and super.users properties.*
> > broker.id - This website is for sale! - broker Resources and
> > Information.<http://broker.id/>
> > broker.id
> > This website is for sale! broker.id is your first and best source for
> all
> > of the information you’re looking for. From general topics to more of
> what
> > you would expect to find here, broker.id has it all. We hope you find
> > what you are searching for!
> >
> >
> >
> > > broker.id - This website is for sale! - broker Resources and
> > > Information.<http://broker.id/>
> > > broker.id
> > > This website is for sale! broker.id is your first and best source for
> > all
> > > of the information you’re looking for. From general topics to more of
> > what
> > > you would expect to find here, broker.id has it all. We hope you find
> > > what you are searching for!
> > >
> > >
> > >
> > >
> > > # ID and basic topic creation
> > > broker.id=1
> > > auto.create.topics.enable=true
> > > delete.topic.enable=true
> > >
> > > # LISTERN Settings
> > > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
> > > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
> > > 172.21.190.176:9093
> > > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
> > > inter.broker.listener.name=INTERNAL
> > > host.name=172.21.190.176
> > >
> > > # Security Settings
> > > ssl.keystore.location=keystore.jks
> > > ssl.keystore.password=password
> > > ssl.key.password=password
> > > ssl.truststore.location=truststore.jks
> > > ssl.truststore.password=password
> > > ssl.keystore.type=JKS
> > > ssl.truststore.type=JKS
> > > security.protocol=SSL
> > > ssl.client.auth=required
> > > allow.everyone.if.no.acl.found=false
> > > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > > super.users=User:CN=Kafka1
> > >
> > > Can you please point out if anything needs to be modified ?
> > >
> > > Many thanks.
> > >
> > > --Darshan
> > >
> > > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > > wrote:
> > >
> > > > If you want to use different interfaces with the same security
> > protocol,
> > > > you can specify listener names. You can then also configure different
> > > > security properties for internal/external if you need.
> > > >
> > > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> > > >
> > > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.
> x.x.x:9093
> > > >
> > > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
> > > >
> > > > inter.broker.listener.name=INTERNAL
> > > >
> > > > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com>
> wrote:
> > > >
> > > > > Hello Darshan
> > > > >
> > > > > Have you tried SSL://0.0.0.0:9093 ?
> > > > >
> > > > > Rajani had suggested something similar to me a week back while I
> was
> > > > > trying to get a ACL based setup.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Wed, May 31, 2017 at 8:58 AM, Darshan <
> > purandare.darshan@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi
> > > > >>
> > > > >> Our Kafka broker has two IPs on two different interfaces.
> > > > >>
> > > > >> eth0 has 172.x.x.x for external leg
> > > > >> eth1 has 1.x.x.x for internal leg
> > > > >>
> > > > >>
> > > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
> > > 1.x.x.x
> > > > >> subnet.
> > > > >>
> > > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then
> Producer
> > > can
> > > > >> producer the message, but Consumer cannot receive the message.
> > > > >>
> > > > >> What value should we use for advertised.listeners so that Producer
> > can
> > > > >> write and Consumers can read ?
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Raghav
> > > > >
> > > >
> > >
> >
>
Re: advertised.listeners
Posted by Joe Hammerman <jh...@squarespace.com.INVALID>.
Hi all,
Is it possible to run mixed mode with PLAINTEXT and SSL with no SASL? What
should port and advertised listeners values in kafka-server.properties be
set in order to configure such an access profile? We want to so in order to
be able to perform healthchecking on the loopback device without having to
negotiate an SSL connection.
Thanks in advance for any assistance anyone can provide,
Joseph Hammerman
On Tue, Apr 3, 2018 at 8:06 PM, Martin Gainty <mg...@hotmail.com> wrote:
> (from MessageBroker consumer)
>
> >tracert MessageProducer
>
> if zk server is found in tracert
>
> then yes you have a MB quorom
>
> FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
> https://stackoverflow.com/questions/46912937/is-it-
> possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broker-and-its-cl
>
> but there is a ZK SASL SASL-Auth exception backdoor in
> zookeeper.properties when
> zookeeper.main_connection_despite_sasl_failure="yes"
>
>
> catch (SaslException e) {
> LOG.warn("Client failed to SASL authenticate: " + e);
> if ((System.getProperty("zookeeper.maintain_connection_despite_sasl_failure")
> != null)
> &&
> (System.getProperty("zookeeper.maintain_connection_
> despite_sasl_failure").equals("yes"))) {
> LOG.warn("Maintaining client connection despite SASL authentication
> failure.");
> } else {
> LOG.warn("Closing client connection due to SASL authentication
> failure.");
> cnxn.close();
> ..
> }
>
>
> does this conform to your findings?
>
> M
> ______________________________________________
>
>
>
> ________________________________
> From: Darshan <pu...@gmail.com>
> Sent: Tuesday, April 3, 2018 6:47 PM
> To: users@kafka.apache.org
> Subject: Re: advertised.listeners
>
> We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
> piece of info ? Thanks.
>
> On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com> wrote:
>
> > tracert MessageBrokerIP
> >
> > do you see ZK server in the trace?
> >
> > if yes then you are running kafka-cluster
> >
> > (ZK does not support mixed mode but there is a backdoor
> > zookeeper.properties config attribute that allows plaintext clients to
> > bypass sasl auth)
> >
> > ?
> >
> > Martin
> > ______________________________________________
> >
> >
> >
> > ________________________________
> > From: Darshan <pu...@gmail.com>
> > Sent: Tuesday, April 3, 2018 5:45 PM
> > To: rajinisivaram@gmail.com
> > Cc: users@kafka.apache.org
> > Subject: Re: advertised.listeners
> >
> > Hi Rajini
> >
> > The above configuration that you mentioned a while back helped me sort
> the
> > issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL and
> > ACLs as well from one of your other posts.
> >
> > I wanted to ask you if it is possible to run Kafka in a mixed security
> > mode? i.e. external producers who are on 172.x.x.x interface can use the
> > SSL to send/receive data from/to our Kafka brokers, but our internal
> > consumer can read/write on PLAINTEXT channel to Kafka.
> >
> > Here is my server.properties, but my internal producer which does not
> have
> > any keystore and truststore is getting the following error:
> > *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
> > correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
> > (org.apache.kafka.clients.NetworkClient)*
> >
> >
> > *server.properties for our Kafka-1,2,3. They are identical except
> > broker.id
> > <http://broker.id> and super.users properties.*
> broker.id - This website is for sale! - broker Resources and
> Information.<http://broker.id/>
> broker.id
> This website is for sale! broker.id is your first and best source for all
> of the information you’re looking for. From general topics to more of what
> you would expect to find here, broker.id has it all. We hope you find
> what you are searching for!
>
>
>
> > broker.id - This website is for sale! - broker Resources and
> > Information.<http://broker.id/>
> > broker.id
> > This website is for sale! broker.id is your first and best source for
> all
> > of the information you’re looking for. From general topics to more of
> what
> > you would expect to find here, broker.id has it all. We hope you find
> > what you are searching for!
> >
> >
> >
> >
> > # ID and basic topic creation
> > broker.id=1
> > auto.create.topics.enable=true
> > delete.topic.enable=true
> >
> > # LISTERN Settings
> > listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
> > advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
> > 172.21.190.176:9093
> > listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
> > inter.broker.listener.name=INTERNAL
> > host.name=172.21.190.176
> >
> > # Security Settings
> > ssl.keystore.location=keystore.jks
> > ssl.keystore.password=password
> > ssl.key.password=password
> > ssl.truststore.location=truststore.jks
> > ssl.truststore.password=password
> > ssl.keystore.type=JKS
> > ssl.truststore.type=JKS
> > security.protocol=SSL
> > ssl.client.auth=required
> > allow.everyone.if.no.acl.found=false
> > authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> > super.users=User:CN=Kafka1
> >
> > Can you please point out if anything needs to be modified ?
> >
> > Many thanks.
> >
> > --Darshan
> >
> > On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <
> rajinisivaram@gmail.com>
> > wrote:
> >
> > > If you want to use different interfaces with the same security
> protocol,
> > > you can specify listener names. You can then also configure different
> > > security properties for internal/external if you need.
> > >
> > > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> > >
> > > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> > >
> > > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
> > >
> > > inter.broker.listener.name=INTERNAL
> > >
> > > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com> wrote:
> > >
> > > > Hello Darshan
> > > >
> > > > Have you tried SSL://0.0.0.0:9093 ?
> > > >
> > > > Rajani had suggested something similar to me a week back while I was
> > > > trying to get a ACL based setup.
> > > >
> > > > Thanks.
> > > >
> > > > On Wed, May 31, 2017 at 8:58 AM, Darshan <
> purandare.darshan@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> Our Kafka broker has two IPs on two different interfaces.
> > > >>
> > > >> eth0 has 172.x.x.x for external leg
> > > >> eth1 has 1.x.x.x for internal leg
> > > >>
> > > >>
> > > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
> > 1.x.x.x
> > > >> subnet.
> > > >>
> > > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then Producer
> > can
> > > >> producer the message, but Consumer cannot receive the message.
> > > >>
> > > >> What value should we use for advertised.listeners so that Producer
> can
> > > >> write and Consumers can read ?
> > > >>
> > > >> Thanks.
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Raghav
> > > >
> > >
> >
>
Re: advertised.listeners
Posted by Martin Gainty <mg...@hotmail.com>.
(from MessageBroker consumer)
>tracert MessageProducer
if zk server is found in tracert
then yes you have a MB quorom
FWIR Mixing PLAINTEXT with SASL-SSL on ZK is not supported
https://stackoverflow.com/questions/46912937/is-it-possible-to-connect-zookeeper-and-kafka-via-sasl-kafka-broker-and-its-cl
but there is a ZK SASL SASL-Auth exception backdoor in zookeeper.properties when
zookeeper.main_connection_despite_sasl_failure="yes"
catch (SaslException e) {
LOG.warn("Client failed to SASL authenticate: " + e);
if ((System.getProperty("zookeeper.maintain_connection_despite_sasl_failure") != null)
&&
(System.getProperty("zookeeper.maintain_connection_despite_sasl_failure").equals("yes"))) {
LOG.warn("Maintaining client connection despite SASL authentication failure.");
} else {
LOG.warn("Closing client connection due to SASL authentication failure.");
cnxn.close();
..
}
does this conform to your findings?
M
______________________________________________
________________________________
From: Darshan <pu...@gmail.com>
Sent: Tuesday, April 3, 2018 6:47 PM
To: users@kafka.apache.org
Subject: Re: advertised.listeners
We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
piece of info ? Thanks.
On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com> wrote:
> tracert MessageBrokerIP
>
> do you see ZK server in the trace?
>
> if yes then you are running kafka-cluster
>
> (ZK does not support mixed mode but there is a backdoor
> zookeeper.properties config attribute that allows plaintext clients to
> bypass sasl auth)
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Darshan <pu...@gmail.com>
> Sent: Tuesday, April 3, 2018 5:45 PM
> To: rajinisivaram@gmail.com
> Cc: users@kafka.apache.org
> Subject: Re: advertised.listeners
>
> Hi Rajini
>
> The above configuration that you mentioned a while back helped me sort the
> issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL and
> ACLs as well from one of your other posts.
>
> I wanted to ask you if it is possible to run Kafka in a mixed security
> mode? i.e. external producers who are on 172.x.x.x interface can use the
> SSL to send/receive data from/to our Kafka brokers, but our internal
> consumer can read/write on PLAINTEXT channel to Kafka.
>
> Here is my server.properties, but my internal producer which does not have
> any keystore and truststore is getting the following error:
> *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
> correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
> (org.apache.kafka.clients.NetworkClient)*
>
>
> *server.properties for our Kafka-1,2,3. They are identical except
> broker.id
> <http://broker.id> and super.users properties.*
broker.id - This website is for sale! - broker Resources and Information.<http://broker.id/>
broker.id
This website is for sale! broker.id is your first and best source for all of the information you’re looking for. From general topics to more of what you would expect to find here, broker.id has it all. We hope you find what you are searching for!
> broker.id - This website is for sale! - broker Resources and
> Information.<http://broker.id/>
> broker.id
> This website is for sale! broker.id is your first and best source for all
> of the information you’re looking for. From general topics to more of what
> you would expect to find here, broker.id has it all. We hope you find
> what you are searching for!
>
>
>
>
> # ID and basic topic creation
> broker.id=1
> auto.create.topics.enable=true
> delete.topic.enable=true
>
> # LISTERN Settings
> listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
> advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
> 172.21.190.176:9093
> listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
> inter.broker.listener.name=INTERNAL
> host.name=172.21.190.176
>
> # Security Settings
> ssl.keystore.location=keystore.jks
> ssl.keystore.password=password
> ssl.key.password=password
> ssl.truststore.location=truststore.jks
> ssl.truststore.password=password
> ssl.keystore.type=JKS
> ssl.truststore.type=JKS
> security.protocol=SSL
> ssl.client.auth=required
> allow.everyone.if.no.acl.found=false
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> super.users=User:CN=Kafka1
>
> Can you please point out if anything needs to be modified ?
>
> Many thanks.
>
> --Darshan
>
> On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > If you want to use different interfaces with the same security protocol,
> > you can specify listener names. You can then also configure different
> > security properties for internal/external if you need.
> >
> > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> >
> > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> >
> > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
> >
> > inter.broker.listener.name=INTERNAL
> >
> > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com> wrote:
> >
> > > Hello Darshan
> > >
> > > Have you tried SSL://0.0.0.0:9093 ?
> > >
> > > Rajani had suggested something similar to me a week back while I was
> > > trying to get a ACL based setup.
> > >
> > > Thanks.
> > >
> > > On Wed, May 31, 2017 at 8:58 AM, Darshan <pu...@gmail.com>
> > > wrote:
> > >
> > >> Hi
> > >>
> > >> Our Kafka broker has two IPs on two different interfaces.
> > >>
> > >> eth0 has 172.x.x.x for external leg
> > >> eth1 has 1.x.x.x for internal leg
> > >>
> > >>
> > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
> 1.x.x.x
> > >> subnet.
> > >>
> > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then Producer
> can
> > >> producer the message, but Consumer cannot receive the message.
> > >>
> > >> What value should we use for advertised.listeners so that Producer can
> > >> write and Consumers can read ?
> > >>
> > >> Thanks.
> > >>
> > >
> > >
> > >
> > > --
> > > Raghav
> > >
> >
>
Re: advertised.listeners
Posted by Darshan <pu...@gmail.com>.
We are using 0.10.2.1 and ZK 3.4.9. Can something be derived from this
piece of info ? Thanks.
On Tue, Apr 3, 2018 at 3:13 PM, Martin Gainty <mg...@hotmail.com> wrote:
> tracert MessageBrokerIP
>
> do you see ZK server in the trace?
>
> if yes then you are running kafka-cluster
>
> (ZK does not support mixed mode but there is a backdoor
> zookeeper.properties config attribute that allows plaintext clients to
> bypass sasl auth)
>
> ?
>
> Martin
> ______________________________________________
>
>
>
> ________________________________
> From: Darshan <pu...@gmail.com>
> Sent: Tuesday, April 3, 2018 5:45 PM
> To: rajinisivaram@gmail.com
> Cc: users@kafka.apache.org
> Subject: Re: advertised.listeners
>
> Hi Rajini
>
> The above configuration that you mentioned a while back helped me sort the
> issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL and
> ACLs as well from one of your other posts.
>
> I wanted to ask you if it is possible to run Kafka in a mixed security
> mode? i.e. external producers who are on 172.x.x.x interface can use the
> SSL to send/receive data from/to our Kafka brokers, but our internal
> consumer can read/write on PLAINTEXT channel to Kafka.
>
> Here is my server.properties, but my internal producer which does not have
> any keystore and truststore is getting the following error:
> *[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
> correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
> (org.apache.kafka.clients.NetworkClient)*
>
>
> *server.properties for our Kafka-1,2,3. They are identical except
> broker.id
> <http://broker.id> and super.users properties.*
> broker.id - This website is for sale! - broker Resources and
> Information.<http://broker.id/>
> broker.id
> This website is for sale! broker.id is your first and best source for all
> of the information you’re looking for. From general topics to more of what
> you would expect to find here, broker.id has it all. We hope you find
> what you are searching for!
>
>
>
>
> # ID and basic topic creation
> broker.id=1
> auto.create.topics.enable=true
> delete.topic.enable=true
>
> # LISTERN Settings
> listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
> advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
> 172.21.190.176:9093
> listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
> inter.broker.listener.name=INTERNAL
> host.name=172.21.190.176
>
> # Security Settings
> ssl.keystore.location=keystore.jks
> ssl.keystore.password=password
> ssl.key.password=password
> ssl.truststore.location=truststore.jks
> ssl.truststore.password=password
> ssl.keystore.type=JKS
> ssl.truststore.type=JKS
> security.protocol=SSL
> ssl.client.auth=required
> allow.everyone.if.no.acl.found=false
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
> super.users=User:CN=Kafka1
>
> Can you please point out if anything needs to be modified ?
>
> Many thanks.
>
> --Darshan
>
> On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > If you want to use different interfaces with the same security protocol,
> > you can specify listener names. You can then also configure different
> > security properties for internal/external if you need.
> >
> > listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> >
> > advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
> >
> > listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
> >
> > inter.broker.listener.name=INTERNAL
> >
> > On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com> wrote:
> >
> > > Hello Darshan
> > >
> > > Have you tried SSL://0.0.0.0:9093 ?
> > >
> > > Rajani had suggested something similar to me a week back while I was
> > > trying to get a ACL based setup.
> > >
> > > Thanks.
> > >
> > > On Wed, May 31, 2017 at 8:58 AM, Darshan <pu...@gmail.com>
> > > wrote:
> > >
> > >> Hi
> > >>
> > >> Our Kafka broker has two IPs on two different interfaces.
> > >>
> > >> eth0 has 172.x.x.x for external leg
> > >> eth1 has 1.x.x.x for internal leg
> > >>
> > >>
> > >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on
> 1.x.x.x
> > >> subnet.
> > >>
> > >> If we use advertised.listeners=SSL://172.x.x.x:9093, then Producer
> can
> > >> producer the message, but Consumer cannot receive the message.
> > >>
> > >> What value should we use for advertised.listeners so that Producer can
> > >> write and Consumers can read ?
> > >>
> > >> Thanks.
> > >>
> > >
> > >
> > >
> > > --
> > > Raghav
> > >
> >
>
Re: advertised.listeners
Posted by Martin Gainty <mg...@hotmail.com>.
tracert MessageBrokerIP
do you see ZK server in the trace?
if yes then you are running kafka-cluster
(ZK does not support mixed mode but there is a backdoor zookeeper.properties config attribute that allows plaintext clients to bypass sasl auth)
?
Martin
______________________________________________
________________________________
From: Darshan <pu...@gmail.com>
Sent: Tuesday, April 3, 2018 5:45 PM
To: rajinisivaram@gmail.com
Cc: users@kafka.apache.org
Subject: Re: advertised.listeners
Hi Rajini
The above configuration that you mentioned a while back helped me sort the
issue of listeners and I was also able to run Kafka 0.10.2.1 with SSL and
ACLs as well from one of your other posts.
I wanted to ask you if it is possible to run Kafka in a mixed security
mode? i.e. external producers who are on 172.x.x.x interface can use the
SSL to send/receive data from/to our Kafka brokers, but our internal
consumer can read/write on PLAINTEXT channel to Kafka.
Here is my server.properties, but my internal producer which does not have
any keystore and truststore is getting the following error:
*[2018-04-03 21:21:04,378] WARN Error while fetching metadata with
correlation id 1 : {Topic4006=TOPIC_AUTHORIZATION_FAILED}
(org.apache.kafka.clients.NetworkClient)*
*server.properties for our Kafka-1,2,3. They are identical except broker.id
<http://broker.id> and super.users properties.*
broker.id - This website is for sale! - broker Resources and Information.<http://broker.id/>
broker.id
This website is for sale! broker.id is your first and best source for all of the information you’re looking for. From general topics to more of what you would expect to find here, broker.id has it all. We hope you find what you are searching for!
# ID and basic topic creation
broker.id=1
auto.create.topics.enable=true
delete.topic.enable=true
# LISTERN Settings
listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://172.21.190.176:9093
advertised.listeners=INTERNAL://1.1.1.165:9092,EXTERNAL://
172.21.190.176:9093
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SSL
inter.broker.listener.name=INTERNAL
host.name=172.21.190.176
# Security Settings
ssl.keystore.location=keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=truststore.jks
ssl.truststore.password=password
ssl.keystore.type=JKS
ssl.truststore.type=JKS
security.protocol=SSL
ssl.client.auth=required
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=Kafka1
Can you please point out if anything needs to be modified ?
Many thanks.
--Darshan
On Wed, May 31, 2017 at 11:31 AM, Rajini Sivaram <ra...@gmail.com>
wrote:
> If you want to use different interfaces with the same security protocol,
> you can specify listener names. You can then also configure different
> security properties for internal/external if you need.
>
> listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
>
> advertised.listeners=INTERNAL://1.x.x.x:9092,EXTERNAL://172.x.x.x:9093
>
> listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
>
> inter.broker.listener.name=INTERNAL
>
> On Wed, May 31, 2017 at 6:22 PM, Raghav <ra...@gmail.com> wrote:
>
> > Hello Darshan
> >
> > Have you tried SSL://0.0.0.0:9093 ?
> >
> > Rajani had suggested something similar to me a week back while I was
> > trying to get a ACL based setup.
> >
> > Thanks.
> >
> > On Wed, May 31, 2017 at 8:58 AM, Darshan <pu...@gmail.com>
> > wrote:
> >
> >> Hi
> >>
> >> Our Kafka broker has two IPs on two different interfaces.
> >>
> >> eth0 has 172.x.x.x for external leg
> >> eth1 has 1.x.x.x for internal leg
> >>
> >>
> >> Kafka Producer is on 172.x.x.x subnet, and Kafka Consumer is on 1.x.x.x
> >> subnet.
> >>
> >> If we use advertised.listeners=SSL://172.x.x.x:9093, then Producer can
> >> producer the message, but Consumer cannot receive the message.
> >>
> >> What value should we use for advertised.listeners so that Producer can
> >> write and Consumers can read ?
> >>
> >> Thanks.
> >>
> >
> >
> >
> > --
> > Raghav
> >
>