You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by IT Consultant <0b...@gmail.com> on 2017/03/01 20:45:51 UTC
Securing Multi-Node single broker kafka instance
Hi Team ,
Can you please help me understand ,
1. How can i secure multi-node (3 machine) single broker (1 broker ) Apache
Kafka deployment secure using SSL ?
i tried to follow instructions here but found pretty confusing .
https://www.confluent.io/blog/apache-kafka-security-authoriz
ation-authentication-encryption/
http://docs.confluent.io/2.0.0/kafka/security.html
Currently , i have kafka running on 3 different machines .
2. How do i make them talk to each other over SSL ?
3. How do i make zookeeper talk to each other and brokers?
Requesting your help .
Thanks in advance.
Re: Securing Multi-Node single broker kafka instance
Posted by IT Consultant <0b...@gmail.com>.
Hi Harsha ,
Just looked at the URL you shared .
I have ensured that zookeeper.properties file is same across all nodes .
Just like it's shown here .
As i stated earlier , its working well for quite sometime .
tickTime=2000
dataDir=/var/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888
Generation of a key and certificate is enough or should i do anything
on zookeeper front to make it work with
kafka brokers ?
Am i missing anything here?
On Thu, Mar 2, 2017 at 3:08 AM, IT Consultant <0b...@gmail.com>
wrote:
> Sure Harsha . I shall follow recommended method .
>
> However , i would like to add to the discussion that current deployment
> worked just fine .
>
> People were using it for quite sometime with no security .
>
> Do i need to create topics and all again if am enabling security ?
>
> On Thu, Mar 2, 2017 at 3:03 AM, Harsha <ka...@harsha.io> wrote:
>
>> Here is the recommended way to setup a 3-node Kafka cluster. Its always
>> recommended to keep zookeeper nodes on different set of nodes than the one
>> you are running Kafka. To go with your current 3-node installation.
>> 1. Install 3-node zookeeper make sure they are forming the quorum (
>> https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html)
>> 2. Install apache kafka binaries on all 3 nodes.
>> 3. Make sure you keep the same zookeeper.connect in server.properties on
>> all 3 nodes for your kafka broker.
>> 4. Start Kafka brokers
>> 5. For sanity check, make sure you create a topic with 3-replication
>> factor and see if you can produce & consume messages
>>
>> Before stepping into security make sure your non-secure Kafka cluster
>> works ok. Once you’ve a stable & working cluster
>> follow instructions in the doc to enable SSL.
>>
>> -Harsha
>>
>> On Mar 1, 2017, 1:08 PM -0800, IT Consultant <0b...@gmail.com>,
>> wrote:
>> > Hi Harsha ,
>> >
>> > Thanks a lot .
>> >
>> > Let me explain where am i stuck ,
>> >
>> > i have three machines on which i am running apache kafka with single
>> broker
>> > but zookeeper of each machine is configured with other machine.
>> >
>> > Example : node1=zk1,zk2,zk3
>> > node2=zk1,zk2,zk3
>> > node3=zk1,zk2,zk3
>> >
>> > This is done for HA .
>> >
>> > Now i need to secure this deployment using SSL .
>> >
>> > *Things tried so far :*
>> >
>> > Create a key and certificate for each of these nodes and configure
>> broker
>> > according to the documentation .
>> >
>> > However , i see following error when i run console producer and consumer
>> > with client certificate or client properties file .
>> >
>> > WARN Error while fetching metadata for topic
>> >
>> >
>> > How do i make each broker work with other broker ?
>> > How do i generate and store certificate for this ? because online
>> document
>> > seems to be confusing for me.
>> > How do i make zookeepers sync with each other and behave as earlier ?
>> >
>> >
>> >
>> > On Thu, Mar 2, 2017 at 2:25 AM, Harsha Chintalapani <ka...@harsha.io>
>> wrote:
>> >
>> > > For inter broker communication over SSL all you need is to add
>> > > security.inter.broker.protocol to SSL.
>> > > "How do i make zookeeper talk to each other and brokers?"
>> > > Not sure I understand the question. You need to make sure zookeeper
>> hosts
>> > > and port are reachable from your broker nodes.
>> > > -Harsha
>> > >
>> > > On Wed, Mar 1, 2017 at 12:45 PM IT Consultant <
>> 0binarybuddha1@gmail.com
>> > > wrote:
>> > >
>> > > > Hi Team ,
>> > > >
>> > > > Can you please help me understand ,
>> > > >
>> > > > 1. How can i secure multi-node (3 machine) single broker (1 broker )
>> > > Apache
>> > > > Kafka deployment secure using SSL ?
>> > > >
>> > > > i tried to follow instructions here but found pretty confusing .
>> > > >
>> > > > https://www.confluent.io/blog/apache-kafka-security-authoriz
>> > > > ation-authentication-encryption/
>> > > >
>> > > > http://docs.confluent.io/2.0.0/kafka/security.html
>> > > >
>> > > > Currently , i have kafka running on 3 different machines .
>> > > > 2. How do i make them talk to each other over SSL ?
>> > > > 3. How do i make zookeeper talk to each other and brokers?
>> > > >
>> > > > Requesting your help .
>> > > >
>> > > > Thanks in advance.
>> > > >
>> > >
>>
>
>
Re: Securing Multi-Node single broker kafka instance
Posted by IT Consultant <0b...@gmail.com>.
Sure Harsha . I shall follow recommended method .
However , i would like to add to the discussion that current deployment
worked just fine .
People were using it for quite sometime with no security .
Do i need to create topics and all again if am enabling security ?
On Thu, Mar 2, 2017 at 3:03 AM, Harsha <ka...@harsha.io> wrote:
> Here is the recommended way to setup a 3-node Kafka cluster. Its always
> recommended to keep zookeeper nodes on different set of nodes than the one
> you are running Kafka. To go with your current 3-node installation.
> 1. Install 3-node zookeeper make sure they are forming the quorum (
> https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html)
> 2. Install apache kafka binaries on all 3 nodes.
> 3. Make sure you keep the same zookeeper.connect in server.properties on
> all 3 nodes for your kafka broker.
> 4. Start Kafka brokers
> 5. For sanity check, make sure you create a topic with 3-replication
> factor and see if you can produce & consume messages
>
> Before stepping into security make sure your non-secure Kafka cluster
> works ok. Once you’ve a stable & working cluster
> follow instructions in the doc to enable SSL.
>
> -Harsha
>
> On Mar 1, 2017, 1:08 PM -0800, IT Consultant <0b...@gmail.com>,
> wrote:
> > Hi Harsha ,
> >
> > Thanks a lot .
> >
> > Let me explain where am i stuck ,
> >
> > i have three machines on which i am running apache kafka with single
> broker
> > but zookeeper of each machine is configured with other machine.
> >
> > Example : node1=zk1,zk2,zk3
> > node2=zk1,zk2,zk3
> > node3=zk1,zk2,zk3
> >
> > This is done for HA .
> >
> > Now i need to secure this deployment using SSL .
> >
> > *Things tried so far :*
> >
> > Create a key and certificate for each of these nodes and configure broker
> > according to the documentation .
> >
> > However , i see following error when i run console producer and consumer
> > with client certificate or client properties file .
> >
> > WARN Error while fetching metadata for topic
> >
> >
> > How do i make each broker work with other broker ?
> > How do i generate and store certificate for this ? because online
> document
> > seems to be confusing for me.
> > How do i make zookeepers sync with each other and behave as earlier ?
> >
> >
> >
> > On Thu, Mar 2, 2017 at 2:25 AM, Harsha Chintalapani <ka...@harsha.io>
> wrote:
> >
> > > For inter broker communication over SSL all you need is to add
> > > security.inter.broker.protocol to SSL.
> > > "How do i make zookeeper talk to each other and brokers?"
> > > Not sure I understand the question. You need to make sure zookeeper
> hosts
> > > and port are reachable from your broker nodes.
> > > -Harsha
> > >
> > > On Wed, Mar 1, 2017 at 12:45 PM IT Consultant <
> 0binarybuddha1@gmail.com
> > > wrote:
> > >
> > > > Hi Team ,
> > > >
> > > > Can you please help me understand ,
> > > >
> > > > 1. How can i secure multi-node (3 machine) single broker (1 broker )
> > > Apache
> > > > Kafka deployment secure using SSL ?
> > > >
> > > > i tried to follow instructions here but found pretty confusing .
> > > >
> > > > https://www.confluent.io/blog/apache-kafka-security-authoriz
> > > > ation-authentication-encryption/
> > > >
> > > > http://docs.confluent.io/2.0.0/kafka/security.html
> > > >
> > > > Currently , i have kafka running on 3 different machines .
> > > > 2. How do i make them talk to each other over SSL ?
> > > > 3. How do i make zookeeper talk to each other and brokers?
> > > >
> > > > Requesting your help .
> > > >
> > > > Thanks in advance.
> > > >
> > >
>
Re: Securing Multi-Node single broker kafka instance
Posted by Harsha <ka...@harsha.io>.
Here is the recommended way to setup a 3-node Kafka cluster. Its always recommended to keep zookeeper nodes on different set of nodes than the one you are running Kafka. To go with your current 3-node installation.
1. Install 3-node zookeeper make sure they are forming the quorum (https://zookeeper.apache.org/doc/r3.3.2/zookeeperAdmin.html)
2. Install apache kafka binaries on all 3 nodes.
3. Make sure you keep the same zookeeper.connect in server.properties on all 3 nodes for your kafka broker.
4. Start Kafka brokers
5. For sanity check, make sure you create a topic with 3-replication factor and see if you can produce & consume messages
Before stepping into security make sure your non-secure Kafka cluster works ok. Once you’ve a stable & working cluster
follow instructions in the doc to enable SSL.
-Harsha
On Mar 1, 2017, 1:08 PM -0800, IT Consultant <0b...@gmail.com>, wrote:
> Hi Harsha ,
>
> Thanks a lot .
>
> Let me explain where am i stuck ,
>
> i have three machines on which i am running apache kafka with single broker
> but zookeeper of each machine is configured with other machine.
>
> Example : node1=zk1,zk2,zk3
> node2=zk1,zk2,zk3
> node3=zk1,zk2,zk3
>
> This is done for HA .
>
> Now i need to secure this deployment using SSL .
>
> *Things tried so far :*
>
> Create a key and certificate for each of these nodes and configure broker
> according to the documentation .
>
> However , i see following error when i run console producer and consumer
> with client certificate or client properties file .
>
> WARN Error while fetching metadata for topic
>
>
> How do i make each broker work with other broker ?
> How do i generate and store certificate for this ? because online document
> seems to be confusing for me.
> How do i make zookeepers sync with each other and behave as earlier ?
>
>
>
> On Thu, Mar 2, 2017 at 2:25 AM, Harsha Chintalapani <ka...@harsha.io> wrote:
>
> > For inter broker communication over SSL all you need is to add
> > security.inter.broker.protocol to SSL.
> > "How do i make zookeeper talk to each other and brokers?"
> > Not sure I understand the question. You need to make sure zookeeper hosts
> > and port are reachable from your broker nodes.
> > -Harsha
> >
> > On Wed, Mar 1, 2017 at 12:45 PM IT Consultant <0binarybuddha1@gmail.com
> > wrote:
> >
> > > Hi Team ,
> > >
> > > Can you please help me understand ,
> > >
> > > 1. How can i secure multi-node (3 machine) single broker (1 broker )
> > Apache
> > > Kafka deployment secure using SSL ?
> > >
> > > i tried to follow instructions here but found pretty confusing .
> > >
> > > https://www.confluent.io/blog/apache-kafka-security-authoriz
> > > ation-authentication-encryption/
> > >
> > > http://docs.confluent.io/2.0.0/kafka/security.html
> > >
> > > Currently , i have kafka running on 3 different machines .
> > > 2. How do i make them talk to each other over SSL ?
> > > 3. How do i make zookeeper talk to each other and brokers?
> > >
> > > Requesting your help .
> > >
> > > Thanks in advance.
> > >
> >
Re: Securing Multi-Node single broker kafka instance
Posted by IT Consultant <0b...@gmail.com>.
Hi Harsha ,
Thanks a lot .
Let me explain where am i stuck ,
i have three machines on which i am running apache kafka with single broker
but zookeeper of each machine is configured with other machine.
Example : node1=zk1,zk2,zk3
node2=zk1,zk2,zk3
node3=zk1,zk2,zk3
This is done for HA .
Now i need to secure this deployment using SSL .
*Things tried so far :*
Create a key and certificate for each of these nodes and configure broker
according to the documentation .
However , i see following error when i run console producer and consumer
with client certificate or client properties file .
WARN Error while fetching metadata for topic
How do i make each broker work with other broker ?
How do i generate and store certificate for this ? because online document
seems to be confusing for me.
How do i make zookeepers sync with each other and behave as earlier ?
On Thu, Mar 2, 2017 at 2:25 AM, Harsha Chintalapani <ka...@harsha.io> wrote:
> For inter broker communication over SSL all you need is to add
> security.inter.broker.protocol to SSL.
> "How do i make zookeeper talk to each other and brokers?"
> Not sure I understand the question. You need to make sure zookeeper hosts
> and port are reachable from your broker nodes.
> -Harsha
>
> On Wed, Mar 1, 2017 at 12:45 PM IT Consultant <0b...@gmail.com>
> wrote:
>
> > Hi Team ,
> >
> > Can you please help me understand ,
> >
> > 1. How can i secure multi-node (3 machine) single broker (1 broker )
> Apache
> > Kafka deployment secure using SSL ?
> >
> > i tried to follow instructions here but found pretty confusing .
> >
> > https://www.confluent.io/blog/apache-kafka-security-authoriz
> > ation-authentication-encryption/
> >
> > http://docs.confluent.io/2.0.0/kafka/security.html
> >
> > Currently , i have kafka running on 3 different machines .
> > 2. How do i make them talk to each other over SSL ?
> > 3. How do i make zookeeper talk to each other and brokers?
> >
> > Requesting your help .
> >
> > Thanks in advance.
> >
>
Re: Securing Multi-Node single broker kafka instance
Posted by Harsha Chintalapani <ka...@harsha.io>.
For inter broker communication over SSL all you need is to add
security.inter.broker.protocol to SSL.
"How do i make zookeeper talk to each other and brokers?"
Not sure I understand the question. You need to make sure zookeeper hosts
and port are reachable from your broker nodes.
-Harsha
On Wed, Mar 1, 2017 at 12:45 PM IT Consultant <0b...@gmail.com>
wrote:
> Hi Team ,
>
> Can you please help me understand ,
>
> 1. How can i secure multi-node (3 machine) single broker (1 broker ) Apache
> Kafka deployment secure using SSL ?
>
> i tried to follow instructions here but found pretty confusing .
>
> https://www.confluent.io/blog/apache-kafka-security-authoriz
> ation-authentication-encryption/
>
> http://docs.confluent.io/2.0.0/kafka/security.html
>
> Currently , i have kafka running on 3 different machines .
> 2. How do i make them talk to each other over SSL ?
> 3. How do i make zookeeper talk to each other and brokers?
>
> Requesting your help .
>
> Thanks in advance.
>