Kafka Authenication and Monitoring Tools

[jo...@accenture.com]

Hi,

Which security implementation is imposed by Kafka? What type of authentication is required by Kafka? Which tools would best monitor the performance of Kafka clusters?


Joseph Simmons
Accenture LLC
TGP-CMT Analyst
San Francisco, CA

B: 415.537.4703
C: 951.941.1125


________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. .
______________________________________________________________________________________

www.accenture.com

Re: Kafka Authenication and Monitoring Tools

[Robert Rodgers <rs...@gmail.com>]

> https://cwiki.apache.org/confluence/display/KAFKA/Security

Interesting. 

I'm curious what people are doing in the interim.

We'd been chewing on options for secure communication and mutual authentication like stunnel.  Actually imposing discretionary controls on the activity of clients using this model would require an server-side proxy if not done inside Kafka itself which on a first pass seemed like it would be invasive.

TLS with client certificates takes advantage of the in-place infrastructure (and allows a variety of reporting and options like pinning, ...) as well as making the client mostly self describing (permissions as extensions, mostly as patterns and mostly about topic access); that actually would be nicer for some scenarios than putting the information in Zookeeper.

Re: Kafka Authenication and Monitoring Tools

[Joe Stein <jo...@stealth.ly>]

https://cwiki.apache.org/confluence/display/KAFKA/Security

https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing

https://cwiki.apache.org/confluence/display/KAFKA/System+Tools

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Tue, Jan 14, 2014 at 2:02 PM, <jo...@accenture.com> wrote:

> Hi,
>
> Which security implementation is imposed by Kafka? What type of
> authentication is required by Kafka? Which tools would best monitor the
> performance of Kafka clusters?
>
>
> Joseph Simmons
> Accenture LLC
> TGP-CMT Analyst
> San Francisco, CA
>
> B: 415.537.4703
> C: 951.941.1125
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy. .
>
> ______________________________________________________________________________________
>
> www.accenture.com
>

RE: Kafka Authenication and Monitoring Tools

[jo...@accenture.com]

Thank You Otis, I'll check this out.

-----Original Message-----
From: Otis Gospodnetic [mailto:otis.gospodnetic@gmail.com]
Sent: Tuesday, January 14, 2014 9:12 PM
To: users@kafka.apache.org
Subject: Re: Kafka Authenication and Monitoring Tools

Hi Joseph,

For the monitoring piece there is SPM - see http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/.
 It will monitor your ZooKeeper, too.  Kafka uses ZK, so you may want to keep an eye on it as well.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/


On Tue, Jan 14, 2014 at 2:02 PM, <jo...@accenture.com> wrote:

> Hi,
>
> Which security implementation is imposed by Kafka? What type of
> authentication is required by Kafka? Which tools would best monitor
> the performance of Kafka clusters?
>
>
> Joseph Simmons
> Accenture LLC
> TGP-CMT Analyst
> San Francisco, CA
>
> B: 415.537.4703
> C: 951.941.1125
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you
> have received it in error, please notify the sender immediately and
> delete the original. Any other use of the e-mail by you is prohibited.
> Where allowed by local law, electronic communications with Accenture
> and its affiliates, including e-mail and instant messaging (including
> content), may be scanned by our systems for the purposes of
> information security and assessment of internal compliance with Accenture policy. .
>
> ______________________________________________________________________
> ________________
>
> www.accenture.com
>

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. .
______________________________________________________________________________________

www.accenture.com

Re: Kafka Authenication and Monitoring Tools

[Otis Gospodnetic <ot...@gmail.com>]

Hi Joseph,

For the monitoring piece there is SPM - see
http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/.
 It will monitor your ZooKeeper, too.  Kafka uses ZK, so you may want
to
keep an eye on it as well.

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Tue, Jan 14, 2014 at 2:02 PM, <jo...@accenture.com> wrote:

> Hi,
>
> Which security implementation is imposed by Kafka? What type of
> authentication is required by Kafka? Which tools would best monitor the
> performance of Kafka clusters?
>
>
> Joseph Simmons
> Accenture LLC
> TGP-CMT Analyst
> San Francisco, CA
>
> B: 415.537.4703
> C: 951.941.1125
>
>
> ________________________________
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy. .
>
> ______________________________________________________________________________________
>
> www.accenture.com
>