Security JIRAS

[Jay Kreps <ja...@gmail.com>]

Hey guys,

As promised, I added a tree of JIRAs for the stuff in the security wiki (
https://cwiki.apache.org/confluence/display/KAFKA/Security):

https://issues.apache.org/jira/browse/KAFKA-1682

I tried to break it into reasonably standalone pieces. I think many of the
tickets could actually be done in parallel. Since there were many people
interested in this area this may help parallelize the work a bit.

I added some strawman details on implementation to each ticket. We can
discuss and refine further on the individual tickets.

Please take a look and let me know if this breakdown seems reasonable.

Cheers,

-Jay

Re: Security JIRAS

[Gwen Shapira <gs...@cloudera.com>]

Yes, I think we can focus on Broker to Zookeeper communication only.
At least for initial stage.

Gwen

On Fri, Oct 17, 2014 at 2:10 PM, Todd Palino <tp...@gmail.com> wrote:
> For the moment, consumers still need to write under the /consumers tree.
> Even if they are committing offsets to Kafka instead of ZK, they will need
> to write owner information there when they are balancing. Eventually, you
> are correct, this is going away with the new consumer.
>
> -Todd
>
> On Fri, Oct 17, 2014 at 10:09 AM, Arvind Mani <am...@linkedin.com.invalid>
> wrote:
>
>>
>> I'm looking at Kafka Brokers authentication with ZooKeeper since this
>> looks independent of other tasks.
>>
>> [AM]
>>
>> 1) Is authentication required only between kafka broker and zookeeper? Can
>> we assume "world" read so that consumers don't have to be authenticated (I
>> believe in any case kafka is planning to change in such that consumers
>> don't have to interact with zk)? In this case I assume kafka broker can I
>> think easily create the znode with appropriate acl list - broker can be
>> admin.
>>
>> 2)  Zookeeper supports Kerberos authentication. Zookeeper supports SSL
>> connections (version 3.4 or later) but I don't see an x509 authentication
>> provider. Do we want to support x509 cert based authentication for zk?
>>
>> - Arvind
>>
>>

Re: Security JIRAS

[Todd Palino <tp...@gmail.com>]

For the moment, consumers still need to write under the /consumers tree.
Even if they are committing offsets to Kafka instead of ZK, they will need
to write owner information there when they are balancing. Eventually, you
are correct, this is going away with the new consumer.

-Todd

On Fri, Oct 17, 2014 at 10:09 AM, Arvind Mani <am...@linkedin.com.invalid>
wrote:

>
> I'm looking at Kafka Brokers authentication with ZooKeeper since this
> looks independent of other tasks.
>
> [AM]
>
> 1) Is authentication required only between kafka broker and zookeeper? Can
> we assume "world" read so that consumers don't have to be authenticated (I
> believe in any case kafka is planning to change in such that consumers
> don't have to interact with zk)? In this case I assume kafka broker can I
> think easily create the znode with appropriate acl list - broker can be
> admin.
>
> 2)  Zookeeper supports Kerberos authentication. Zookeeper supports SSL
> connections (version 3.4 or later) but I don't see an x509 authentication
> provider. Do we want to support x509 cert based authentication for zk?
>
> - Arvind
>
>

RE: Security JIRAS

[Arvind Mani <am...@linkedin.com.INVALID>]

I'm looking at Kafka Brokers authentication with ZooKeeper since this
looks independent of other tasks.

[AM] 

1) Is authentication required only between kafka broker and zookeeper? Can we assume "world" read so that consumers don't have to be authenticated (I believe in any case kafka is planning to change in such that consumers don't have to interact with zk)? In this case I assume kafka broker can I think easily create the znode with appropriate acl list - broker can be admin.

2)  Zookeeper supports Kerberos authentication. Zookeeper supports SSL connections (version 3.4 or later) but I don't see an x509 authentication provider. Do we want to support x509 cert based authentication for zk? 

- Arvind

Re: Security JIRAS

[Gwen Shapira <gs...@cloudera.com>]

Wondering the same here :)

I think there are some parallel threads here (SSL is independent of
Kerberos, as far as I can see).

Kerberos work is blocked on
https://issues.apache.org/jira/browse/KAFKA-1683 - "Implement a
"session" concept in the socket server". So there's no point in
picking up other tasks before this is assigned (and at least
designed).

I'm looking at Kafka Brokers authentication with ZooKeeper since this
looks independent of other tasks.

Gwen



On Thu, Oct 16, 2014 at 4:23 PM, Michael Herstine
<mh...@linkedin.com.invalid> wrote:
> Thanks, Jay.
>
> I¹m new to the project, and I¹m wondering how things proceed from hereŠ
> are folks working on these tasks, or do they get assigned, orŠ?
>
> On 10/7/14, 5:15 PM, "Jay Kreps" <ja...@gmail.com> wrote:
>
>>Hey guys,
>>
>>As promised, I added a tree of JIRAs for the stuff in the security wiki (
>>https://cwiki.apache.org/confluence/display/KAFKA/Security):
>>
>>https://issues.apache.org/jira/browse/KAFKA-1682
>>
>>I tried to break it into reasonably standalone pieces. I think many of the
>>tickets could actually be done in parallel. Since there were many people
>>interested in this area this may help parallelize the work a bit.
>>
>>I added some strawman details on implementation to each ticket. We can
>>discuss and refine further on the individual tickets.
>>
>>Please take a look and let me know if this breakdown seems reasonable.
>>
>>Cheers,
>>
>>-Jay
>

Re: Security JIRAS

[Michael Herstine <mh...@linkedin.com.INVALID>]

Thanks, Jay.

I¹m new to the project, and I¹m wondering how things proceed from hereŠ
are folks working on these tasks, or do they get assigned, orŠ?

On 10/7/14, 5:15 PM, "Jay Kreps" <ja...@gmail.com> wrote:

>Hey guys,
>
>As promised, I added a tree of JIRAs for the stuff in the security wiki (
>https://cwiki.apache.org/confluence/display/KAFKA/Security):
>
>https://issues.apache.org/jira/browse/KAFKA-1682
>
>I tried to break it into reasonably standalone pieces. I think many of the
>tickets could actually be done in parallel. Since there were many people
>interested in this area this may help parallelize the work a bit.
>
>I added some strawman details on implementation to each ticket. We can
>discuss and refine further on the individual tickets.
>
>Please take a look and let me know if this breakdown seems reasonable.
>
>Cheers,
>
>-Jay

Re: Security JIRAS

[Jarek Jarcec Cecho <ja...@apache.org>]

I would be strong +1 on that. I’ve seen a lot of regressions on other projects when new functionality cause regressions when running in secure mode.

Jarcec

On Oct 10, 2014, at 9:43 AM, Neha Narkhede <ne...@gmail.com> wrote:

> I'd vote for accepting every major change with the relevant system tests.
> We didn't do this for major features in the past that lead to weak coverage
> and a great deal of work for someone else to add tests for features that
> were done in the past. I'm guilty of this myself :-(
> 
> On Thu, Oct 9, 2014 at 6:45 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> 
>> Added some details on delegation tokens. I hope it at least clarifies
>> some of the scope.
>> I'm working on a more detailed design doc.
>> 
>> On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> Hey Gwen,
>>> 
>>> Your absolutely right about these. I added the ticket for ZK
>> authentication
>>> and Hadoop delegation tokens.
>>> 
>>> For the Hadoop case I actually don't understand Hadoop security very
>> well.
>>> Maybe you could fill in some of the details on what needs to happen for
>>> that to work?
>>> 
>>> For testing, we should probably discuss the best way to test security. I
>>> think this is a fairly critical thing, if we are going to say we have
>>> security we really need to have good tests in place to ensure we do. This
>>> will require some thought. I think we should be able to test TLS fairly
>>> easily using junit integration test that just starts the server and
>>> connects using TLS. For Kerberos though it isn't clear to me how to do
>> good
>>> integration testing since we need a KDC to test against and it isn't
>> clear
>>> how that happens in the test environment except possibly manually (which
>> is
>>> not ideal). How do other projects handle this?
>>> 
>>> -Jay
>>> 
>>> On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira <gs...@cloudera.com>
>> wrote:
>>> 
>>>> I think we need to add:
>>>> 
>>>> * Authentication of Kafka brokers with a secured ZooKeeper
>>>> * Kafka should be able to generate delegation tokens for MapReduce /
>>>> Spark / Yarn jobs.
>>>> * Extend systest framework to allow testing secured kafka
>>>> 
>>>> Gwen
>>>> 
>>>> On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps <ja...@gmail.com> wrote:
>>>>> Hey guys,
>>>>> 
>>>>> As promised, I added a tree of JIRAs for the stuff in the security
>> wiki (
>>>>> https://cwiki.apache.org/confluence/display/KAFKA/Security):
>>>>> 
>>>>> https://issues.apache.org/jira/browse/KAFKA-1682
>>>>> 
>>>>> I tried to break it into reasonably standalone pieces. I think many of
>>>> the
>>>>> tickets could actually be done in parallel. Since there were many
>> people
>>>>> interested in this area this may help parallelize the work a bit.
>>>>> 
>>>>> I added some strawman details on implementation to each ticket. We can
>>>>> discuss and refine further on the individual tickets.
>>>>> 
>>>>> Please take a look and let me know if this breakdown seems reasonable.
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>> -Jay
>>>> 
>> 

Re: Security JIRAS

[Neha Narkhede <ne...@gmail.com>]

I'd vote for accepting every major change with the relevant system tests.
We didn't do this for major features in the past that lead to weak coverage
and a great deal of work for someone else to add tests for features that
were done in the past. I'm guilty of this myself :-(

On Thu, Oct 9, 2014 at 6:45 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Added some details on delegation tokens. I hope it at least clarifies
> some of the scope.
> I'm working on a more detailed design doc.
>
> On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps <ja...@gmail.com> wrote:
> > Hey Gwen,
> >
> > Your absolutely right about these. I added the ticket for ZK
> authentication
> > and Hadoop delegation tokens.
> >
> > For the Hadoop case I actually don't understand Hadoop security very
> well.
> > Maybe you could fill in some of the details on what needs to happen for
> > that to work?
> >
> > For testing, we should probably discuss the best way to test security. I
> > think this is a fairly critical thing, if we are going to say we have
> > security we really need to have good tests in place to ensure we do. This
> > will require some thought. I think we should be able to test TLS fairly
> > easily using junit integration test that just starts the server and
> > connects using TLS. For Kerberos though it isn't clear to me how to do
> good
> > integration testing since we need a KDC to test against and it isn't
> clear
> > how that happens in the test environment except possibly manually (which
> is
> > not ideal). How do other projects handle this?
> >
> > -Jay
> >
> > On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> >
> >> I think we need to add:
> >>
> >> * Authentication of Kafka brokers with a secured ZooKeeper
> >> * Kafka should be able to generate delegation tokens for MapReduce /
> >> Spark / Yarn jobs.
> >> * Extend systest framework to allow testing secured kafka
> >>
> >> Gwen
> >>
> >> On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps <ja...@gmail.com> wrote:
> >> > Hey guys,
> >> >
> >> > As promised, I added a tree of JIRAs for the stuff in the security
> wiki (
> >> > https://cwiki.apache.org/confluence/display/KAFKA/Security):
> >> >
> >> > https://issues.apache.org/jira/browse/KAFKA-1682
> >> >
> >> > I tried to break it into reasonably standalone pieces. I think many of
> >> the
> >> > tickets could actually be done in parallel. Since there were many
> people
> >> > interested in this area this may help parallelize the work a bit.
> >> >
> >> > I added some strawman details on implementation to each ticket. We can
> >> > discuss and refine further on the individual tickets.
> >> >
> >> > Please take a look and let me know if this breakdown seems reasonable.
> >> >
> >> > Cheers,
> >> >
> >> > -Jay
> >>
>

Re: Security JIRAS

[Gwen Shapira <gs...@cloudera.com>]

Added some details on delegation tokens. I hope it at least clarifies
some of the scope.
I'm working on a more detailed design doc.

On Thu, Oct 9, 2014 at 1:44 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hey Gwen,
>
> Your absolutely right about these. I added the ticket for ZK authentication
> and Hadoop delegation tokens.
>
> For the Hadoop case I actually don't understand Hadoop security very well.
> Maybe you could fill in some of the details on what needs to happen for
> that to work?
>
> For testing, we should probably discuss the best way to test security. I
> think this is a fairly critical thing, if we are going to say we have
> security we really need to have good tests in place to ensure we do. This
> will require some thought. I think we should be able to test TLS fairly
> easily using junit integration test that just starts the server and
> connects using TLS. For Kerberos though it isn't clear to me how to do good
> integration testing since we need a KDC to test against and it isn't clear
> how that happens in the test environment except possibly manually (which is
> not ideal). How do other projects handle this?
>
> -Jay
>
> On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
>> I think we need to add:
>>
>> * Authentication of Kafka brokers with a secured ZooKeeper
>> * Kafka should be able to generate delegation tokens for MapReduce /
>> Spark / Yarn jobs.
>> * Extend systest framework to allow testing secured kafka
>>
>> Gwen
>>
>> On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps <ja...@gmail.com> wrote:
>> > Hey guys,
>> >
>> > As promised, I added a tree of JIRAs for the stuff in the security wiki (
>> > https://cwiki.apache.org/confluence/display/KAFKA/Security):
>> >
>> > https://issues.apache.org/jira/browse/KAFKA-1682
>> >
>> > I tried to break it into reasonably standalone pieces. I think many of
>> the
>> > tickets could actually be done in parallel. Since there were many people
>> > interested in this area this may help parallelize the work a bit.
>> >
>> > I added some strawman details on implementation to each ticket. We can
>> > discuss and refine further on the individual tickets.
>> >
>> > Please take a look and let me know if this breakdown seems reasonable.
>> >
>> > Cheers,
>> >
>> > -Jay
>>

Re: Security JIRAS

[Jarek Jarcec Cecho <ja...@apache.org>]

> For Kerberos though it isn't clear to me how to do good
> integration testing since we need a KDC to test against and it isn't clear
> how that happens in the test environment except possibly manually (which is
> not ideal). How do other projects handle this?

Actually it’s not that hard. Hadoop is providing small KDC implementation for testing purpose called MiniKdc. It’s super easy to use, check it out in Sentry:

https://github.com/apache/incubator-sentry/blob/master/sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/thrift/SentryMiniKdcTestcase.java

Jarcec

On Oct 9, 2014, at 1:44 PM, Jay Kreps <ja...@gmail.com> wrote:

> Hey Gwen,
> 
> Your absolutely right about these. I added the ticket for ZK authentication
> and Hadoop delegation tokens.
> 
> For the Hadoop case I actually don't understand Hadoop security very well.
> Maybe you could fill in some of the details on what needs to happen for
> that to work?
> 
> For testing, we should probably discuss the best way to test security. I
> think this is a fairly critical thing, if we are going to say we have
> security we really need to have good tests in place to ensure we do. This
> will require some thought. I think we should be able to test TLS fairly
> easily using junit integration test that just starts the server and
> connects using TLS. For Kerberos though it isn't clear to me how to do good
> integration testing since we need a KDC to test against and it isn't clear
> how that happens in the test environment except possibly manually (which is
> not ideal). How do other projects handle this?
> 
> -Jay
> 
> On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> 
>> I think we need to add:
>> 
>> * Authentication of Kafka brokers with a secured ZooKeeper
>> * Kafka should be able to generate delegation tokens for MapReduce /
>> Spark / Yarn jobs.
>> * Extend systest framework to allow testing secured kafka
>> 
>> Gwen
>> 
>> On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> Hey guys,
>>> 
>>> As promised, I added a tree of JIRAs for the stuff in the security wiki (
>>> https://cwiki.apache.org/confluence/display/KAFKA/Security):
>>> 
>>> https://issues.apache.org/jira/browse/KAFKA-1682
>>> 
>>> I tried to break it into reasonably standalone pieces. I think many of
>> the
>>> tickets could actually be done in parallel. Since there were many people
>>> interested in this area this may help parallelize the work a bit.
>>> 
>>> I added some strawman details on implementation to each ticket. We can
>>> discuss and refine further on the individual tickets.
>>> 
>>> Please take a look and let me know if this breakdown seems reasonable.
>>> 
>>> Cheers,
>>> 
>>> -Jay
>> 

Re: Security JIRAS

[Jay Kreps <ja...@gmail.com>]

Hey Gwen,

Your absolutely right about these. I added the ticket for ZK authentication
and Hadoop delegation tokens.

For the Hadoop case I actually don't understand Hadoop security very well.
Maybe you could fill in some of the details on what needs to happen for
that to work?

For testing, we should probably discuss the best way to test security. I
think this is a fairly critical thing, if we are going to say we have
security we really need to have good tests in place to ensure we do. This
will require some thought. I think we should be able to test TLS fairly
easily using junit integration test that just starts the server and
connects using TLS. For Kerberos though it isn't clear to me how to do good
integration testing since we need a KDC to test against and it isn't clear
how that happens in the test environment except possibly manually (which is
not ideal). How do other projects handle this?

-Jay

On Tue, Oct 7, 2014 at 5:25 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> I think we need to add:
>
> * Authentication of Kafka brokers with a secured ZooKeeper
> * Kafka should be able to generate delegation tokens for MapReduce /
> Spark / Yarn jobs.
> * Extend systest framework to allow testing secured kafka
>
> Gwen
>
> On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps <ja...@gmail.com> wrote:
> > Hey guys,
> >
> > As promised, I added a tree of JIRAs for the stuff in the security wiki (
> > https://cwiki.apache.org/confluence/display/KAFKA/Security):
> >
> > https://issues.apache.org/jira/browse/KAFKA-1682
> >
> > I tried to break it into reasonably standalone pieces. I think many of
> the
> > tickets could actually be done in parallel. Since there were many people
> > interested in this area this may help parallelize the work a bit.
> >
> > I added some strawman details on implementation to each ticket. We can
> > discuss and refine further on the individual tickets.
> >
> > Please take a look and let me know if this breakdown seems reasonable.
> >
> > Cheers,
> >
> > -Jay
>

Re: Security JIRAS

[Gwen Shapira <gs...@cloudera.com>]

I think we need to add:

* Authentication of Kafka brokers with a secured ZooKeeper
* Kafka should be able to generate delegation tokens for MapReduce /
Spark / Yarn jobs.
* Extend systest framework to allow testing secured kafka

Gwen

On Tue, Oct 7, 2014 at 5:15 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hey guys,
>
> As promised, I added a tree of JIRAs for the stuff in the security wiki (
> https://cwiki.apache.org/confluence/display/KAFKA/Security):
>
> https://issues.apache.org/jira/browse/KAFKA-1682
>
> I tried to break it into reasonably standalone pieces. I think many of the
> tickets could actually be done in parallel. Since there were many people
> interested in this area this may help parallelize the work a bit.
>
> I added some strawman details on implementation to each ticket. We can
> discuss and refine further on the individual tickets.
>
> Please take a look and let me know if this breakdown seems reasonable.
>
> Cheers,
>
> -Jay