0.9 security features and release schedule

[Justin Randell <ju...@acquia.com>]

Hi,

We're assessing Kafka for use at Acquia. We run tens of thousands of
customer websites and many internal applications on ~9k AWS EC2
instances.

We're currently weighing the pros and cons of starting with 0.82 plus
custom security, or waiting until the security features land in 0.9.

How likely is Kafka 0.9 to ship in April 2015? (As seen here -
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan.)

How stable is the 0.9 branch? Is it crazy to consider running a 0.9
beta in production?

Are there any existing patch sets against 0.8x that implement security features?

thanks,
Justin

Re: 0.9 security features and release schedule

[Justin Randell <ju...@acquia.com>]

hi,

On Sun, Jan 25, 2015 at 11:06 AM, Jeff Holoman <jh...@cloudera.com> wrote:
> Justin,
>
> I'd be interested to learn specifically what security features you are
> interested in.

we are interested in both simple authentication and per-topic
authorization. we can get started with simple authentication, but will
need per-topic authorization or encrypted message bodies before we can
send some event data over Kafka.

> There is a doc on the wiki here
> https://cwiki.apache.org/confluence/display/KAFKA/Security
>
> As well as the Umbrella JIRA Here:
> https://issues.apache.org/jira/browse/KAFKA-1682

thanks, i've read the security page in the wiki, and i'm following the
jira issues.

> Most security-related JIRAs have a component listed as security
> https://issues.apache.org/jira/browse/KAFKA-1882?jql=project%20%3D%20KAFKA%20AND%20component%20%3D%20security
>
> There is a quite a bit of plumbing work to be done in order for all of the
> security features to come together. There technically isn't a 0.9 branch at
> the moment, all of the development is being applied to trunk and since
> there isn't as of yet even a 0.8.3 branch, I don't think you'd be in a good
> spot trying to pull in a bunch of stuff that isn't complete.

right, our initial PoC work and planning for 0.8.2 and security has
focused on simple wrappers around Kafka 0.8.2.

for example, adding an iptables/security group whitelist. this only
gets us part of the way, as by necessity we run customer code on a
large minority of our productions machines (we host a large number of
high traffic Drupal sites), and that code can legitimately talk to the
internet. those instances should be significant producers connected to
Kafka, but we can't simply trust traffic from those instances.

we tested haproxy in front of Kafka, requiring a client SSL
certificate for any code that talks to haproxy. then, we used
advertised.port and advertised.hostname, and monkey-patched the
posseidon ruby client to speak SSL. this works up to a point, but has
it's own problems.

if we use one Kafka cluster, we have to teach all clients to speak SSL
and offer a valid certificate. or, we teach some clients to ignore
advertised.hostname, and some to honour it, depending which method we
use to trust the traffic.

another option is to mirror traffic from the SSL-wrapped kafka to the
iptables-wrapped Kafka.

i also wrote a simple websocket <-> Kafka bridge using the Shopify
sarama client, and that works nicely but would take some effort to
bring to production readiness, and doesn't have any per-topic
authorization.

we think we can use a some combination of the above to start using
Kafka for a large chunk of our log pipeline needs, but it's far from
ideal compared to simply using built-in security features within
Kafka.

how do others solve these sorts of problems with Kafka?

cheers
Justin


> Just my 2 cents.
>
> Thanks
>
> Jeff
>
>
> On Sun, Jan 25, 2015 at 10:33 AM, Justin Randell <ju...@acquia.com>
> wrote:
>
>> Hi,
>>
>> We're assessing Kafka for use at Acquia. We run tens of thousands of
>> customer websites and many internal applications on ~9k AWS EC2
>> instances.
>>
>> We're currently weighing the pros and cons of starting with 0.82 plus
>> custom security, or waiting until the security features land in 0.9.
>>
>> How likely is Kafka 0.9 to ship in April 2015? (As seen here -
>> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan.)
>>
>> How stable is the 0.9 branch? Is it crazy to consider running a 0.9
>> beta in production?
>>
>> Are there any existing patch sets against 0.8x that implement security
>> features?
>>
>> thanks,
>> Justin
>>
>
>
>
> --
> Jeff Holoman
> Systems Engineer

Re: 0.9 security features and release schedule

[Jeff Holoman <jh...@cloudera.com>]

Justin,

I'd be interested to learn specifically what security features you are
interested in.

There is a doc on the wiki here
https://cwiki.apache.org/confluence/display/KAFKA/Security

As well as the Umbrella JIRA Here:
https://issues.apache.org/jira/browse/KAFKA-1682

Most security-related JIRAs have a component listed as security
https://issues.apache.org/jira/browse/KAFKA-1882?jql=project%20%3D%20KAFKA%20AND%20component%20%3D%20security

There is a quite a bit of plumbing work to be done in order for all of the
security features to come together. There technically isn't a 0.9 branch at
the moment, all of the development is being applied to trunk and since
there isn't as of yet even a 0.8.3 branch, I don't think you'd be in a good
spot trying to pull in a bunch of stuff that isn't complete.

Just my 2 cents.

Thanks

Jeff


On Sun, Jan 25, 2015 at 10:33 AM, Justin Randell <ju...@acquia.com>
wrote:

> Hi,
>
> We're assessing Kafka for use at Acquia. We run tens of thousands of
> customer websites and many internal applications on ~9k AWS EC2
> instances.
>
> We're currently weighing the pros and cons of starting with 0.82 plus
> custom security, or waiting until the security features land in 0.9.
>
> How likely is Kafka 0.9 to ship in April 2015? (As seen here -
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan.)
>
> How stable is the 0.9 branch? Is it crazy to consider running a 0.9
> beta in production?
>
> Are there any existing patch sets against 0.8x that implement security
> features?
>
> thanks,
> Justin
>



-- 
Jeff Holoman
Systems Engineer