You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2016/10/31 05:56:08 UTC
Re: [EXTERNAL] kafka
Hi Team,
Can someone pls help me with below requested information ?
Does apache apex have any inbuilt kafka input operator to read from Kafka 0.9 secure kafka topics?
Thanks a lot.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Monday, October 24, 2016 at 2:29 PM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: [EXTERNAL] kafka
Hi,
Do we have any kaka operator readily available to consume messages from secure kafka topics in kafka 0.9 clusters?
Thanks a lot.
Regards,
Raja.
Re: [EXTERNAL] kafka
Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks a lot for the response Pramod.
Our kafka environment is NOT Kerberized! But the kafka topic is secure via SSL certificates!
Only hosts with valid SSL certificates can only connect! Any thoughts in this regard will be really helpful!!
Thanks a lot!!
Regards,
Raja.
From: Pramod Immaneni <pr...@datatorrent.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Tuesday, November 1, 2016 at 3:24 AM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: Re: [EXTERNAL] kafka
Hi Raja,
Yes we have had success reading from kerberized secure Kafka. If I remember correctly we had to set a few attributes like container jvm options in Apex. Is your setup Kerberos secured Kafka?
Thanks
On Oct 30, 2016, at 10:56 PM, Raja.Aravapalli <Ra...@target.com>> wrote:
Hi Team,
Can someone pls help me with below requested information ?
Does apache apex have any inbuilt kafka input operator to read from Kafka 0.9 secure kafka topics?
Thanks a lot.
Regards,
Raja.
From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, October 24, 2016 at 2:29 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: [EXTERNAL] kafka
Hi,
Do we have any kaka operator readily available to consume messages from secure kafka topics in kafka 0.9 clusters?
Thanks a lot.
Regards,
Raja.
Re: [EXTERNAL] kafka
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,
Yes we have had success reading from kerberized secure Kafka. If I remember
correctly we had to set a few attributes like container jvm options in
Apex. Is your setup Kerberos secured Kafka?
Thanks
On Oct 30, 2016, at 10:56 PM, Raja.Aravapalli <Ra...@target.com>
wrote:
Hi Team,
Can someone pls help me with below requested information ?
Does apache apex have any inbuilt kafka input operator to read from Kafka
0.9 secure kafka topics?
Thanks a lot.
Regards,
Raja.
*From: *"Raja.Aravapalli" <Ra...@target.com>
*Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
*Date: *Monday, October 24, 2016 at 2:29 PM
*To: *"users@apex.apache.org" <us...@apex.apache.org>
*Subject: *[EXTERNAL] kafka
Hi,
Do we have any kaka operator readily available to consume messages from
secure kafka topics in kafka 0.9 clusters?
Thanks a lot.
Regards,
Raja.
Re: [EXTERNAL] kafka
Posted by Thomas Weise <th...@apache.org>.
IMO this should be added here:
http://apex.apache.org/docs/malhar/operators/kafkaInputOperator/
On Tue, Nov 1, 2016 at 5:24 PM, hsy541@gmail.com <hs...@gmail.com> wrote:
> Hey Raja,
>
> The setup for secure kafka input operator is not easy. You can follow
> these steps.
> 1. Follow kafka document to setup your brokers properly (
> https://kafka.apache.org/090/documentation.html#security_overview)
> 2. You have to manually create the client JAAS file (
> https://kafka.apache.org/090/documentation.html#security_overview)
> a sample file would look like this:
> KafkaClient {
>
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka_client.keytab"
> principal="kafka-client-1@EXAMPLE.COM";
> };
>
> 3. On the operator you have to set the attribute JVM_OPTS
> -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
>
> 4. On the operator you have to set the consumerProperties, for example
>
> set dt.operator.$your_operator_name.{consumerProps.security.protocol} to
> SASL or other security function you use
> set dt.operator.$your_operator_name.{consumerPrope.sasl.
> kerberos.service.name} to kafka
>
>
> Hope this would help you!
>
>
> Regards,
> Siyuan
>
>
>
> On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>>
>> Hi Team,
>>
>>
>>
>> Can someone pls help me with below requested information ?
>>
>>
>>
>> Does apache apex have any inbuilt kafka input operator to read from Kafka
>> 0.9 secure kafka topics?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> Regards,
>>
>> Raja.
>>
>>
>>
>> *From: *"Raja.Aravapalli" <Ra...@target.com>
>> *Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
>> *Date: *Monday, October 24, 2016 at 2:29 PM
>> *To: *"users@apex.apache.org" <us...@apex.apache.org>
>> *Subject: *[EXTERNAL] kafka
>>
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>> Do we have any kaka operator readily available to consume messages from
>> secure kafka topics in kafka 0.9 clusters?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>>
>>
>> Regards,
>>
>> Raja.
>>
>
>
Re: [EXTERNAL] kafka
Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Hey Raja,
The setup for secure kafka input operator is not easy. You can follow these
steps.
1. Follow kafka document to setup your brokers properly (
https://kafka.apache.org/090/documentation.html#security_overview)
2. You have to manually create the client JAAS file (
https://kafka.apache.org/090/documentation.html#security_overview)
a sample file would look like this:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.keytab"
principal="kafka-client-1@EXAMPLE.COM";
};
3. On the operator you have to set the attribute JVM_OPTS
-Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
4. On the operator you have to set the consumerProperties, for example
set dt.operator.$your_operator_name.{consumerProps.security.protocol} to
SASL or other security function you use
set dt.operator.$your_operator_name.{consumerPrope.
sasl.kerberos.service.name} to kafka
Hope this would help you!
Regards,
Siyuan
On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli <
Raja.Aravapalli@target.com> wrote:
>
>
> Hi Team,
>
>
>
> Can someone pls help me with below requested information ?
>
>
>
> Does apache apex have any inbuilt kafka input operator to read from Kafka
> 0.9 secure kafka topics?
>
>
>
> Thanks a lot.
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *"Raja.Aravapalli" <Ra...@target.com>
> *Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
> *Date: *Monday, October 24, 2016 at 2:29 PM
> *To: *"users@apex.apache.org" <us...@apex.apache.org>
> *Subject: *[EXTERNAL] kafka
>
>
>
>
>
> Hi,
>
>
>
> Do we have any kaka operator readily available to consume messages from
> secure kafka topics in kafka 0.9 clusters?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>