You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Manoj Murumkar <ma...@gmail.com> on 2017/09/06 06:08:10 UTC
Re: Re: Using ACLs without Kerberos
Thanks, SASL setup for ZK does the trick.
Now, I am continuing the security setup with schema registry. I am
configuring it to use same PLAINTEXT method for now.
listeners=http://0.0.0.0:8081
kafkastore.connection.url=localhost:2181
kafkastore.topic=_schemas
debug=true
# Security specific params
kafkastore.bootstrap.servers=SASL_PLAINTEXT://localhost:9092
#kafkastore.bootstrap.servers=SASL_PLAINTEXT://or1010050208015:9092,SASL_PLAINTEXT://or1010050208016:9092,SASL_PLAINTEXT://or1010050208017:9092,SASL_PLAINTEXT://or1010050208018:9092
kafkastore.init.timeout.ms=10000
kafkastore.security.protocol=SASL_PLAINTEXT
zookeeper.set.acl=true
kafkastore.sasl.mechanism=PLAIN
kafkastore.group.id=schema-registry-consumer-group
I have followed instructions (modified for my use case and latest version
of confluent) from here:
https://www.confluent.io/blog/securing-confluent-schema-registry-apache-kafka/
When I try to start schema registry, it fails with following error:
[2017-09-06 06:05:23,790] ERROR Server died unexpectedly:
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:52)
java.lang.IllegalArgumentException: Unable to subscribe to the Kafka topic
_schemas backing this data store. Topic may not exist.
at
io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread.<init>(KafkaStoreReaderThread.java:130)
However, I have enabled schema registry user to produce/consume messages
for _schemas topic and authorizer debug messages show the same:
[2017-09-06 06:05:21,784] DEBUG operation = Read on resource =
Topic:_schemas from host = 127.0.0.1 is Allow based on acl = User:s
chemaregistry has Allow permission for operations: All from hosts: *
(kafka.authorizer.logger)
[2017-09-06 06:05:21,784] DEBUG Principal = User:schemaregistry is Allowed
Operation = Describe from host = 127.0.0.1 on resource
= Topic:_schemas (kafka.authorizer.logger)
[2017-09-06 06:05:22,788] DEBUG operation = Read on resource =
Topic:_schemas from host = 127.0.0.1 is Allow based on acl =
User:schemaregistry has Allow permission for operations: All from hosts: *
(kafka.authorizer.logger)
[2017-09-06 06:05:22,788] DEBUG Principal = User:schemaregistry is Allowed
Operation = Describe from host = 127.0.0.1 on resource = Topic:_schemas
(kafka.authorizer.logger)
Any ideas of what might be happening here?
Here're contents of my JAAS file that I use in the startup script:
[root@or1010050208015 kafka]# tail -1 /bin/schema-registry-start
exec $(dirname $0)/schema-registry-run-class ${EXTRA_ARGS}
-Djava.security.auth.login.config=/etc/schema-registry/schema_registry_client_jaas.properties
io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain "$@"
[root@or1010050208015 kafka]# cat
/etc/schema-registry/schema_registry_client_jaas.properties
/* Kafka Client */
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="schemaregistry"
password="schemaRegistry";
};
/* Zookeeper client */
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="zk_admin"
password="zk_admin_secret";
};
On Thu, Aug 31, 2017 at 1:35 PM, Shrikant Patel <SP...@pdxinc.com> wrote:
> Yes, as per my understanding for now the only way to secure ZK is SASL.
> https://issues.apache.org/jira/browse/ZOOKEEPER-2125 once released ZK
> could also be secured using SSL.
>
> Also remember not use any OS user, anyone user on n\w who can connect to
> ZK host:port will be able to modify the ACLs.
>
> - Shri
>
> -----Original Message-----
> From: Manoj Murumkar [mailto:manoj.murumkar@gmail.com]
> Sent: Thursday, August 31, 2017 11:03 AM
> To: users@kafka.apache.org
> Subject: [EXTERNAL] Re: Using ACLs without Kerberos
>
> ***** Notice: This email was received from an external source *****
>
> >> Current kafka-acls.sh script directly contacts zookeeper to create ACLs.
> Any OS user who got access to zookeeper can create ACLs for any Kafka
> principal.
>
> Thanks for that point. Appreciate it.
>
> On Thu, Aug 31, 2017 at 8:29 AM, Manikumar <ma...@gmail.com>
> wrote:
>
> > There is no correlation between OS user and Kafka Principal/Username.
> > Here user name refers to the principal associated with the kafka
> > communication channel (Kerberos Principal, SASL/Plain username, Scram
> > username, SSL
> > certificate)
> >
> > Current kafka-acls.sh script directly contacts zookeeper to create ACLs.
> > Any OS user who got access to zookeeper can create ACLs for any Kafka
> > principal.
> > This can be restricted by enabling Zk Kerberos authentication.
> > http://kafka.apache.org/documentation/#zk_authz
> >
> > From Kafka 0.11.0.0, We can create ACLs using Java Admin API.
> >
> > On Thu, Aug 31, 2017 at 7:56 PM, Manoj Murumkar
> > <ma...@gmail.com>
> > wrote:
> >
> > > Right, I am. Just to be clear, I am using kafka-acl script to
> > define/remove
> > > ACLs as a non-super user and it just works fine. I had expected it
> > > to
> > work
> > > only for super users and not for regular users ('nex37045' is a
> > > normal user).
> > >
> > > [nex37045@or1010051029033 ~]$ kafka-acls --authorizer
> > > kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
> > > zookeeper.connect=localhost:2181 --list Current ACLs for resource
> > > `Topic:test`:
> > > User:nex37045 has Deny permission for operations: Write from
> hosts:
> > *
> > > User:skumarmu has Allow permission for operations: Write from
> hosts:
> > *
> > >
> > > [nex37045@or1010051029033 ~]$
> > > [nex37045@or1010051029033 ~]$ kafka-acls --authorizer
> > > kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
> > > zookeeper.connect=localhost:2181 --remove --allow-principal
> > User:skumarmu
> > > --operation Write --topic test
> > > Are you sure you want to remove ACLs:
> > > User:skumarmu has Allow permission for operations: Write from
> > hosts: *
> > > from resource `Topic:test`? (y/n)
> > > y
> > > Current ACLs for resource `Topic:test`:
> > > User:nex37045 has Deny permission for operations: Write from
> hosts:
> > *
> > >
> > > [nex37045@or1010051029033 ~]$ kafka-acls --authorizer
> > > kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
> > > zookeeper.connect=localhost:2181 --list Current ACLs for resource
> > > `Topic:test`:
> > > User:nex37045 has Deny permission for operations: Write from
> hosts:
> > *
> > >
> > >
> > >
> > > On Thu, Aug 31, 2017 at 7:16 AM, Manikumar
> > > <ma...@gmail.com>
> > > wrote:
> > >
> > > > Looks like you are already using SASL/PLAIN mechanism. Kafka
> > > > supports
> > > SASL
> > > > authentication framework.
> > > > KAFKA SASL supports GSSAPI (Kerberos), PLAIN or SCRAM mechanisms.
> > > > you
> > can
> > > > enable SSL encryption also
> > > >
> > > > http://kafka.apache.org/documentation.html#security
> > > >
> > > >
> > > > On Thu, Aug 31, 2017 at 7:28 PM, Manoj Murumkar <
> > > manoj.murumkar@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Manikumar. I am testing the setup documented here:
> > > > > https://developer.ibm.com/opentech/2017/05/31/kafka-
> > acls-in-practice/
> > > > > (SASL_PLAINTEXT).
> > > > >
> > > > > I haven't setup any authentication for the tests. Thinking about
> > > > > it, authentication is a must have for authorization (so, kafka
> > > > > knows
> > who's
> > > > > making resource request), right?
> > > > >
> > > > > On Wed, Aug 30, 2017 at 11:39 PM, Manikumar <
> > manikumar.reddy@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Kafka default authorizer is used with secure authenticated
> > > > > > channels (SSL,SASL,SCRAM).
> > > > > > For plain text (non-secure) channels, the principal will be
> > > > > > always ANONYMOUS. Here you can authorize by ip-address.
> > > > > >
> > > > > > It's adviced to run on secure channels. you can try SASL/PLAIN
> > > > > > or
> > > SCRAM
> > > > > > mechanisms with/without SSL.
> > > > > > Pl, check Kafka docs for security considerations.
> > > > > >
> > > > > > >> I was testing if a non-admin principal (OS user) can modify
> > > > > (add/remove)
> > > > > > ACLs and
> > > > > > it seems like it's possible.
> > > > > > How are you trying? Can you give more details about your setup?
> > What
> > > > > > authentication mechanism, etc.?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 31, 2017 at 2:49 AM, Manoj Murumkar <
> > > > > manoj.murumkar@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > We are evaluating how to put authorization in place for
> > > > > > > Kafka
> > > (around
> > > > > > > topics, mostly). Is it a good idea to do this without
> > > > > > > Kerberos? I
> > > was
> > > > > > > testing if a non-admin principal (OS user) can modify
> > (add/remove)
> > > > ACLs
> > > > > > and
> > > > > > > it seems like it's possible. If this is right behavior, it's
> > > insecure
> > > > > and
> > > > > > > unusable. What do you guys think?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Manoj
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> This e-mail and its contents (to include attachments) are the property of
> National Health Systems, Inc., its subsidiaries and affiliates, including
> but not limited to Rx.com Community Healthcare Network, Inc. and its
> subsidiaries, and may contain confidential and proprietary or privileged
> information. If you are not the intended recipient of this e-mail, you are
> hereby notified that any unauthorized disclosure, copying, or distribution
> of this e-mail or of its attachments, or the taking of any unauthorized
> action based on information contained herein is strictly prohibited.
> Unauthorized use of information contained herein may subject you to civil
> and criminal prosecution and penalties. If you are not the intended
> recipient, please immediately notify the sender by telephone at
> 800-433-5719 or return e-mail and permanently delete the original e-mail.
>
Re: Re: Using ACLs without Kerberos
Posted by Manoj Murumkar <ma...@gmail.com>.
This is fixed as well, btw. My ZK config wasn't done properly.
On Tue, Sep 5, 2017 at 11:08 PM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Thanks, SASL setup for ZK does the trick.
>
> Now, I am continuing the security setup with schema registry. I am
> configuring it to use same PLAINTEXT method for now.
>
> listeners=http://0.0.0.0:8081
> kafkastore.connection.url=localhost:2181
> kafkastore.topic=_schemas
> debug=true
> # Security specific params
> kafkastore.bootstrap.servers=SASL_PLAINTEXT://localhost:9092
> #kafkastore.bootstrap.servers=SASL_PLAINTEXT://or1010050208015:9092,SASL_
> PLAINTEXT://or1010050208016:9092,SASL_PLAINTEXT://
> or1010050208017:9092,SASL_PLAINTEXT://or1010050208018:9092
> kafkastore.init.timeout.ms=10000
> kafkastore.security.protocol=SASL_PLAINTEXT
> zookeeper.set.acl=true
> kafkastore.sasl.mechanism=PLAIN
> kafkastore.group.id=schema-registry-consumer-group
>
> I have followed instructions (modified for my use case and latest version
> of confluent) from here: https://www.confluent.io/blog/
> securing-confluent-schema-registry-apache-kafka/
>
> When I try to start schema registry, it fails with following error:
>
> [2017-09-06 06:05:23,790] ERROR Server died unexpectedly:
> (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:52)
> java.lang.IllegalArgumentException: Unable to subscribe to the Kafka
> topic _schemas backing this data store. Topic may not exist.
> at io.confluent.kafka.schemaregistry.storage.
> KafkaStoreReaderThread.<init>(KafkaStoreReaderThread.java:130)
>
>
> However, I have enabled schema registry user to produce/consume messages
> for _schemas topic and authorizer debug messages show the same:
>
> [2017-09-06 06:05:21,784] DEBUG operation = Read on resource =
> Topic:_schemas from host = 127.0.0.1 is Allow based on acl = User:s
> chemaregistry has Allow permission for operations: All from hosts: *
> (kafka.authorizer.logger)
> [2017-09-06 06:05:21,784] DEBUG Principal = User:schemaregistry is Allowed
> Operation = Describe from host = 127.0.0.1 on resource
> = Topic:_schemas (kafka.authorizer.logger)
> [2017-09-06 06:05:22,788] DEBUG operation = Read on resource =
> Topic:_schemas from host = 127.0.0.1 is Allow based on acl =
> User:schemaregistry has Allow permission for operations: All from hosts: *
> (kafka.authorizer.logger)
> [2017-09-06 06:05:22,788] DEBUG Principal = User:schemaregistry is Allowed
> Operation = Describe from host = 127.0.0.1 on resource = Topic:_schemas
> (kafka.authorizer.logger)
>
> Any ideas of what might be happening here?
>
> Here're contents of my JAAS file that I use in the startup script:
>
>
> [root@or1010050208015 kafka]# tail -1 /bin/schema-registry-start
> exec $(dirname $0)/schema-registry-run-class ${EXTRA_ARGS}
> -Djava.security.auth.login.config=/etc/schema-registry/
> schema_registry_client_jaas.properties io.confluent.kafka.
> schemaregistry.rest.SchemaRegistryMain "$@"
>
>
> [root@or1010050208015 kafka]# cat /etc/schema-registry/schema_
> registry_client_jaas.properties
> /* Kafka Client */
> KafkaClient {
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="schemaregistry"
> password="schemaRegistry";
> };
>
> /* Zookeeper client */
> Client {
> org.apache.zookeeper.server.auth.DigestLoginModule required
> username="zk_admin"
> password="zk_admin_secret";
> };
>
>
> On Thu, Aug 31, 2017 at 1:35 PM, Shrikant Patel <SP...@pdxinc.com> wrote:
>
>> Yes, as per my understanding for now the only way to secure ZK is SASL.
>> https://issues.apache.org/jira/browse/ZOOKEEPER-2125 once released ZK
>> could also be secured using SSL.
>>
>> Also remember not use any OS user, anyone user on n\w who can connect to
>> ZK host:port will be able to modify the ACLs.
>>
>> - Shri
>>
>> -----Original Message-----
>> From: Manoj Murumkar [mailto:manoj.murumkar@gmail.com]
>> Sent: Thursday, August 31, 2017 11:03 AM
>> To: users@kafka.apache.org
>> Subject: [EXTERNAL] Re: Using ACLs without Kerberos
>>
>> ***** Notice: This email was received from an external source *****
>>
>> >> Current kafka-acls.sh script directly contacts zookeeper to create
>> ACLs.
>> Any OS user who got access to zookeeper can create ACLs for any Kafka
>> principal.
>>
>> Thanks for that point. Appreciate it.
>>
>> On Thu, Aug 31, 2017 at 8:29 AM, Manikumar <ma...@gmail.com>
>> wrote:
>>
>> > There is no correlation between OS user and Kafka Principal/Username.
>> > Here user name refers to the principal associated with the kafka
>> > communication channel (Kerberos Principal, SASL/Plain username, Scram
>> > username, SSL
>> > certificate)
>> >
>> > Current kafka-acls.sh script directly contacts zookeeper to create ACLs.
>> > Any OS user who got access to zookeeper can create ACLs for any Kafka
>> > principal.
>> > This can be restricted by enabling Zk Kerberos authentication.
>> > http://kafka.apache.org/documentation/#zk_authz
>> >
>> > From Kafka 0.11.0.0, We can create ACLs using Java Admin API.
>> >
>> > On Thu, Aug 31, 2017 at 7:56 PM, Manoj Murumkar
>> > <ma...@gmail.com>
>> > wrote:
>> >
>> > > Right, I am. Just to be clear, I am using kafka-acl script to
>> > define/remove
>> > > ACLs as a non-super user and it just works fine. I had expected it
>> > > to
>> > work
>> > > only for super users and not for regular users ('nex37045' is a
>> > > normal user).
>> > >
>> > > [nex37045@or1010051029033 ~]$ kafka-acls --authorizer
>> > > kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
>> > > zookeeper.connect=localhost:2181 --list Current ACLs for resource
>> > > `Topic:test`:
>> > > User:nex37045 has Deny permission for operations: Write from
>> hosts:
>> > *
>> > > User:skumarmu has Allow permission for operations: Write from
>> hosts:
>> > *
>> > >
>> > > [nex37045@or1010051029033 ~]$
>> > > [nex37045@or1010051029033 ~]$ kafka-acls --authorizer
>> > > kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
>> > > zookeeper.connect=localhost:2181 --remove --allow-principal
>> > User:skumarmu
>> > > --operation Write --topic test
>> > > Are you sure you want to remove ACLs:
>> > > User:skumarmu has Allow permission for operations: Write from
>> > hosts: *
>> > > from resource `Topic:test`? (y/n)
>> > > y
>> > > Current ACLs for resource `Topic:test`:
>> > > User:nex37045 has Deny permission for operations: Write from
>> hosts:
>> > *
>> > >
>> > > [nex37045@or1010051029033 ~]$ kafka-acls --authorizer
>> > > kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
>> > > zookeeper.connect=localhost:2181 --list Current ACLs for resource
>> > > `Topic:test`:
>> > > User:nex37045 has Deny permission for operations: Write from
>> hosts:
>> > *
>> > >
>> > >
>> > >
>> > > On Thu, Aug 31, 2017 at 7:16 AM, Manikumar
>> > > <ma...@gmail.com>
>> > > wrote:
>> > >
>> > > > Looks like you are already using SASL/PLAIN mechanism. Kafka
>> > > > supports
>> > > SASL
>> > > > authentication framework.
>> > > > KAFKA SASL supports GSSAPI (Kerberos), PLAIN or SCRAM mechanisms.
>> > > > you
>> > can
>> > > > enable SSL encryption also
>> > > >
>> > > > http://kafka.apache.org/documentation.html#security
>> > > >
>> > > >
>> > > > On Thu, Aug 31, 2017 at 7:28 PM, Manoj Murumkar <
>> > > manoj.murumkar@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Thanks Manikumar. I am testing the setup documented here:
>> > > > > https://developer.ibm.com/opentech/2017/05/31/kafka-
>> > acls-in-practice/
>> > > > > (SASL_PLAINTEXT).
>> > > > >
>> > > > > I haven't setup any authentication for the tests. Thinking about
>> > > > > it, authentication is a must have for authorization (so, kafka
>> > > > > knows
>> > who's
>> > > > > making resource request), right?
>> > > > >
>> > > > > On Wed, Aug 30, 2017 at 11:39 PM, Manikumar <
>> > manikumar.reddy@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Kafka default authorizer is used with secure authenticated
>> > > > > > channels (SSL,SASL,SCRAM).
>> > > > > > For plain text (non-secure) channels, the principal will be
>> > > > > > always ANONYMOUS. Here you can authorize by ip-address.
>> > > > > >
>> > > > > > It's adviced to run on secure channels. you can try SASL/PLAIN
>> > > > > > or
>> > > SCRAM
>> > > > > > mechanisms with/without SSL.
>> > > > > > Pl, check Kafka docs for security considerations.
>> > > > > >
>> > > > > > >> I was testing if a non-admin principal (OS user) can modify
>> > > > > (add/remove)
>> > > > > > ACLs and
>> > > > > > it seems like it's possible.
>> > > > > > How are you trying? Can you give more details about your setup?
>> > What
>> > > > > > authentication mechanism, etc.?
>> > > > > >
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > >
>> > > > > > On Thu, Aug 31, 2017 at 2:49 AM, Manoj Murumkar <
>> > > > > manoj.murumkar@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > We are evaluating how to put authorization in place for
>> > > > > > > Kafka
>> > > (around
>> > > > > > > topics, mostly). Is it a good idea to do this without
>> > > > > > > Kerberos? I
>> > > was
>> > > > > > > testing if a non-admin principal (OS user) can modify
>> > (add/remove)
>> > > > ACLs
>> > > > > > and
>> > > > > > > it seems like it's possible. If this is right behavior, it's
>> > > insecure
>> > > > > and
>> > > > > > > unusable. What do you guys think?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > >
>> > > > > > > Manoj
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> This e-mail and its contents (to include attachments) are the property of
>> National Health Systems, Inc., its subsidiaries and affiliates, including
>> but not limited to Rx.com Community Healthcare Network, Inc. and its
>> subsidiaries, and may contain confidential and proprietary or privileged
>> information. If you are not the intended recipient of this e-mail, you are
>> hereby notified that any unauthorized disclosure, copying, or distribution
>> of this e-mail or of its attachments, or the taking of any unauthorized
>> action based on information contained herein is strictly prohibited.
>> Unauthorized use of information contained herein may subject you to civil
>> and criminal prosecution and penalties. If you are not the intended
>> recipient, please immediately notify the sender by telephone at
>> 800-433-5719 or return e-mail and permanently delete the original e-mail.
>>
>
>