You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Fergal Somers <fe...@workday.com> on 2013/04/23 19:02:42 UTC
Securing Kafka
Hi
We are planning to use Kafka, but like others on this list we have a need
to be able to secure communication. The approaches people have suggested on
this list are:
- Encrypt the messages at the producer (e.g
http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication
)
- Add SSL to Kafka protocol -
http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E
Adding SSL support to Kafka, probably means adding SSLEngine support the
the nio socket handling (
https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ).
I don't think there are any immediate plans to provide this, but it's
potentially something that Kafka would support in the future?
In theory this is something we could look at, but we would need to go
further. We also need to separate producers from consumers. The aim would
be to ensure that a Kafka producer couldn't also act as a consumer.
Essentially producers can write to Kafka, but not read.
>From looking at the Kafka source, achieving producer/consumer separation
looks to me like it would be quite a change to the Kafka server (0.7). So
are there any plans in the (near) future in this area (producer / consumer
separation) ?
Message encryption (at the application layer) would allow us to achieve
both aims of securing communication and separating consumers from
producers. Producers get the public cert (so they can encrypt messages as
they place them on the bus). Only consumers get the private cert - so only
they can decrypt messages consumed. This seems like something we can do
ourselves - I just wanted to sanity check the approach with this group.
Cheers,
Fergal.
Re: Securing Kafka
Posted by Matt Wise <ma...@nextdoor.com>.
The other issue with this model is that you're Kafka servers are available on the internet ... so anyone else can push data into them. At least, if you are running in a cross-cloud/cross-datacenter environment. We're struggling with the same design issues right now.
What we've hoped for is that in 0.8.x, Kafka would allow our producers to connect to ANY Kafka server and submit data, and have that data dynamically routed to the right servers. This way we could put the Kafka servers behind an ELB, throw up Stunnel clients on our producers and Stunnel servers on the Kafka machines. This would offload the SSL encryption and authentication to Stunnel, and allow Kafka to concentrate on what its good at.
For now though, that doesn't seem possible. It looks like we may end up going down the Flume route, because its easier to encrypt and authenticate the data streams through Flume. :/
--Matt
On Apr 23, 2013, at 11:02 AM, Jason Rosenberg <jb...@squareup.com> wrote:
> Yes,
>
> I think encryption at the message level is a workable solution, as long as
> you don't care about exposing the meta data that goes with it (e.g. topic
> names, kafka broker/zk server locations, etc.).
>
> Jason
>
>
> On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers
> <fe...@workday.com>wrote:
>
>> Hi
>>
>> We are planning to use Kafka, but like others on this list we have a need
>> to be able to secure communication. The approaches people have suggested on
>> this list are:
>>
>> - Encrypt the messages at the producer (e.g
>>
>> http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication
>> )
>> - Add SSL to Kafka protocol -
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E
>>
>> Adding SSL support to Kafka, probably means adding SSLEngine support the
>> the nio socket handling (
>> https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ).
>> I don't think there are any immediate plans to provide this, but it's
>> potentially something that Kafka would support in the future?
>>
>> In theory this is something we could look at, but we would need to go
>> further. We also need to separate producers from consumers. The aim would
>> be to ensure that a Kafka producer couldn't also act as a consumer.
>> Essentially producers can write to Kafka, but not read.
>>
>> From looking at the Kafka source, achieving producer/consumer separation
>> looks to me like it would be quite a change to the Kafka server (0.7). So
>> are there any plans in the (near) future in this area (producer / consumer
>> separation) ?
>>
>> Message encryption (at the application layer) would allow us to achieve
>> both aims of securing communication and separating consumers from
>> producers. Producers get the public cert (so they can encrypt messages as
>> they place them on the bus). Only consumers get the private cert - so only
>> they can decrypt messages consumed. This seems like something we can do
>> ourselves - I just wanted to sanity check the approach with this group.
>>
>> Cheers,
>>
>> Fergal.
>>
Re: Securing Kafka
Posted by Fergal Somers <fe...@workday.com>.
Yes securing the content of log messages at rest is important to us - which
favors message encryption.
Thanks for the responses.
Fergal.
On Tue, Apr 23, 2013 at 7:31 PM, Chris Curtin <cu...@gmail.com>wrote:
> Also keep in mind that anything done at the transport (SSL for example)
> layer won't solve your 'at rest' problems.
>
> All messages are written to disk, so unless the broker does some encryption
> logic you haven't solved the data visibility issues.
>
> I also think this should be a producer/consumer problem not a Broker. Keep
> the Brokers as fast as possible (thus NIO/kernel space activities etc.) and
> push the cost to the producers and consumers.
>
> Chris
>
>
> On Tue, Apr 23, 2013 at 2:02 PM, Jason Rosenberg <jb...@squareup.com> wrote:
>
> > Yes,
> >
> > I think encryption at the message level is a workable solution, as long
> as
> > you don't care about exposing the meta data that goes with it (e.g. topic
> > names, kafka broker/zk server locations, etc.).
> >
> > Jason
> >
> >
> > On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers
> > <fe...@workday.com>wrote:
> >
> > > Hi
> > >
> > > We are planning to use Kafka, but like others on this list we have a
> need
> > > to be able to secure communication. The approaches people have
> suggested
> > on
> > > this list are:
> > >
> > > - Encrypt the messages at the producer (e.g
> > >
> > >
> >
> http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication
> > > )
> > > - Add SSL to Kafka protocol -
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E
> > >
> > > Adding SSL support to Kafka, probably means adding SSLEngine support
> the
> > > the nio socket handling (
> > >
> https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ
> > ).
> > > I don't think there are any immediate plans to provide this, but it's
> > > potentially something that Kafka would support in the future?
> > >
> > > In theory this is something we could look at, but we would need to go
> > > further. We also need to separate producers from consumers. The aim
> would
> > > be to ensure that a Kafka producer couldn't also act as a consumer.
> > > Essentially producers can write to Kafka, but not read.
> > >
> > > From looking at the Kafka source, achieving producer/consumer
> separation
> > > looks to me like it would be quite a change to the Kafka server (0.7).
> So
> > > are there any plans in the (near) future in this area (producer /
> > consumer
> > > separation) ?
> > >
> > > Message encryption (at the application layer) would allow us to achieve
> > > both aims of securing communication and separating consumers from
> > > producers. Producers get the public cert (so they can encrypt messages
> as
> > > they place them on the bus). Only consumers get the private cert - so
> > only
> > > they can decrypt messages consumed. This seems like something we can do
> > > ourselves - I just wanted to sanity check the approach with this group.
> > >
> > > Cheers,
> > >
> > > Fergal.
> > >
> >
>
--
Fergal Somers
Architect Integrations
(Skype) fergal.somers.workday
fergal.somers@workday.com
<http://www.workday.com/signature-uk>
Re: Securing Kafka
Posted by Chris Curtin <cu...@gmail.com>.
Also keep in mind that anything done at the transport (SSL for example)
layer won't solve your 'at rest' problems.
All messages are written to disk, so unless the broker does some encryption
logic you haven't solved the data visibility issues.
I also think this should be a producer/consumer problem not a Broker. Keep
the Brokers as fast as possible (thus NIO/kernel space activities etc.) and
push the cost to the producers and consumers.
Chris
On Tue, Apr 23, 2013 at 2:02 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Yes,
>
> I think encryption at the message level is a workable solution, as long as
> you don't care about exposing the meta data that goes with it (e.g. topic
> names, kafka broker/zk server locations, etc.).
>
> Jason
>
>
> On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers
> <fe...@workday.com>wrote:
>
> > Hi
> >
> > We are planning to use Kafka, but like others on this list we have a need
> > to be able to secure communication. The approaches people have suggested
> on
> > this list are:
> >
> > - Encrypt the messages at the producer (e.g
> >
> >
> http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication
> > )
> > - Add SSL to Kafka protocol -
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E
> >
> > Adding SSL support to Kafka, probably means adding SSLEngine support the
> > the nio socket handling (
> > https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ
> ).
> > I don't think there are any immediate plans to provide this, but it's
> > potentially something that Kafka would support in the future?
> >
> > In theory this is something we could look at, but we would need to go
> > further. We also need to separate producers from consumers. The aim would
> > be to ensure that a Kafka producer couldn't also act as a consumer.
> > Essentially producers can write to Kafka, but not read.
> >
> > From looking at the Kafka source, achieving producer/consumer separation
> > looks to me like it would be quite a change to the Kafka server (0.7). So
> > are there any plans in the (near) future in this area (producer /
> consumer
> > separation) ?
> >
> > Message encryption (at the application layer) would allow us to achieve
> > both aims of securing communication and separating consumers from
> > producers. Producers get the public cert (so they can encrypt messages as
> > they place them on the bus). Only consumers get the private cert - so
> only
> > they can decrypt messages consumed. This seems like something we can do
> > ourselves - I just wanted to sanity check the approach with this group.
> >
> > Cheers,
> >
> > Fergal.
> >
>
Re: Securing Kafka
Posted by Jason Rosenberg <jb...@squareup.com>.
Yes,
I think encryption at the message level is a workable solution, as long as
you don't care about exposing the meta data that goes with it (e.g. topic
names, kafka broker/zk server locations, etc.).
Jason
On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers
<fe...@workday.com>wrote:
> Hi
>
> We are planning to use Kafka, but like others on this list we have a need
> to be able to secure communication. The approaches people have suggested on
> this list are:
>
> - Encrypt the messages at the producer (e.g
>
> http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+communication
> )
> - Add SSL to Kafka protocol -
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201304.mbox/ajax/%3CCAA%2BBczQ_dMXUTNndSu4d%2B6aRo%3DSLiFa4iGMu_78OWKub_CTScw%40mail.gmail.com%3E
>
> Adding SSL support to Kafka, probably means adding SSLEngine support the
> the nio socket handling (
> https://groups.google.com/forum/#!msg/kafka-dev/ZmJrB_plu1I/_9cmGlLCSVEJ).
> I don't think there are any immediate plans to provide this, but it's
> potentially something that Kafka would support in the future?
>
> In theory this is something we could look at, but we would need to go
> further. We also need to separate producers from consumers. The aim would
> be to ensure that a Kafka producer couldn't also act as a consumer.
> Essentially producers can write to Kafka, but not read.
>
> From looking at the Kafka source, achieving producer/consumer separation
> looks to me like it would be quite a change to the Kafka server (0.7). So
> are there any plans in the (near) future in this area (producer / consumer
> separation) ?
>
> Message encryption (at the application layer) would allow us to achieve
> both aims of securing communication and separating consumers from
> producers. Producers get the public cert (so they can encrypt messages as
> they place them on the bus). Only consumers get the private cert - so only
> they can decrypt messages consumed. This seems like something we can do
> ourselves - I just wanted to sanity check the approach with this group.
>
> Cheers,
>
> Fergal.
>