You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2016/10/24 08:59:40 UTC

kafka

Hi,

Do we have any kaka operator readily available to consume messages from secure kafka topics in kafka 0.9 clusters?

Thanks a lot.


Regards,
Raja.

Re: kafka

Posted by Chaitanya Chebolu <ch...@datatorrent.com>.
Hi Raja,

  Please refer to the below location for Kafka 0.9 Input Operator:
https://github.com/apache/apex-malhar/tree/master/kafka/src/main/java/org/apache/apex/malhar/kafka

  Please have a look here
<https://github.com/DataTorrent/examples/tree/master/tutorials/kafka> for
example application.

Regards,
Chaitanya

On Mon, Oct 24, 2016 at 2:29 PM, Raja.Aravapalli <Raja.Aravapalli@target.com
> wrote:

>
>
> Hi,
>
>
>
> Do we have any kaka operator readily available to consume messages from
> secure kafka topics in kafka 0.9 clusters?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>

Re: [EXTERNAL] kafka

Posted by "Raja.Aravapalli" <Ra...@target.com>.
Thanks a lot for the response Pramod.

Our kafka environment is NOT Kerberized! But the kafka topic is secure via SSL certificates!

Only hosts with valid SSL certificates can only connect! Any thoughts in this regard will be really helpful!!

Thanks a lot!!


Regards,
Raja.

From: Pramod Immaneni <pr...@datatorrent.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Tuesday, November 1, 2016 at 3:24 AM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: Re: [EXTERNAL] kafka

Hi Raja,

Yes we have had success reading from kerberized secure Kafka. If I remember correctly we had to set a few attributes like container jvm options in Apex. Is your setup Kerberos secured Kafka?

Thanks

On Oct 30, 2016, at 10:56 PM, Raja.Aravapalli <Ra...@target.com>> wrote:

Hi Team,

Can someone pls help me with below requested information ?

Does apache apex have any inbuilt kafka input operator to read from Kafka 0.9 secure kafka topics?

Thanks a lot.

Regards,
Raja.

From: "Raja.Aravapalli" <Ra...@target.com>>
Reply-To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Date: Monday, October 24, 2016 at 2:29 PM
To: "users@apex.apache.org<ma...@apex.apache.org>" <us...@apex.apache.org>>
Subject: [EXTERNAL] kafka


Hi,

Do we have any kaka operator readily available to consume messages from secure kafka topics in kafka 0.9 clusters?

Thanks a lot.


Regards,
Raja.

Re: [EXTERNAL] kafka

Posted by Pramod Immaneni <pr...@datatorrent.com>.
Hi Raja,

Yes we have had success reading from kerberized secure Kafka. If I remember
correctly we had to set a few attributes like container jvm options in
Apex. Is your setup Kerberos secured Kafka?

Thanks

On Oct 30, 2016, at 10:56 PM, Raja.Aravapalli <Ra...@target.com>
wrote:



Hi Team,



Can someone pls help me with below requested information ?



Does apache apex have any inbuilt kafka input operator to read from Kafka
0.9 secure kafka topics?



Thanks a lot.



Regards,

Raja.



*From: *"Raja.Aravapalli" <Ra...@target.com>
*Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
*Date: *Monday, October 24, 2016 at 2:29 PM
*To: *"users@apex.apache.org" <us...@apex.apache.org>
*Subject: *[EXTERNAL] kafka





Hi,



Do we have any kaka operator readily available to consume messages from
secure kafka topics in kafka 0.9 clusters?



Thanks a lot.





Regards,

Raja.

Re: [EXTERNAL] kafka

Posted by Thomas Weise <th...@apache.org>.
IMO this should be added here:

http://apex.apache.org/docs/malhar/operators/kafkaInputOperator/


On Tue, Nov 1, 2016 at 5:24 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> Hey Raja,
>
> The setup for secure kafka input operator is not easy. You can follow
> these steps.
> 1. Follow kafka document to setup your brokers properly (
> https://kafka.apache.org/090/documentation.html#security_overview)
> 2. You have to manually create the client JAAS file (
> https://kafka.apache.org/090/documentation.html#security_overview)
>    a sample file would look like this:
>      KafkaClient {
>
>         com.sun.security.auth.module.Krb5LoginModule required
>         useKeyTab=true
>         storeKey=true
>         keyTab="/etc/security/keytabs/kafka_client.keytab"
>         principal="kafka-client-1@EXAMPLE.COM";
>     };
>
> 3. On the operator you have to set the attribute JVM_OPTS
>  -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
>
> 4. On the operator you have to set the consumerProperties, for example
>
> set dt.operator.$your_operator_name.{consumerProps.security.protocol}  to
>  SASL or other security function you use
> set dt.operator.$your_operator_name.{consumerPrope.sasl.
> kerberos.service.name} to kafka
>
>
> Hope this would help you!
>
>
> Regards,
> Siyuan
>
>
>
> On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>>
>>
>> Hi Team,
>>
>>
>>
>> Can someone pls help me with below requested information ?
>>
>>
>>
>> Does apache apex have any inbuilt kafka input operator to read from Kafka
>> 0.9 secure kafka topics?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> Regards,
>>
>> Raja.
>>
>>
>>
>> *From: *"Raja.Aravapalli" <Ra...@target.com>
>> *Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
>> *Date: *Monday, October 24, 2016 at 2:29 PM
>> *To: *"users@apex.apache.org" <us...@apex.apache.org>
>> *Subject: *[EXTERNAL] kafka
>>
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>> Do we have any kaka operator readily available to consume messages from
>> secure kafka topics in kafka 0.9 clusters?
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>>
>>
>> Regards,
>>
>> Raja.
>>
>
>

Re: [EXTERNAL] kafka

Posted by "hsy541@gmail.com" <hs...@gmail.com>.
Hey Raja,

The setup for secure kafka input operator is not easy. You can follow these
steps.
1. Follow kafka document to setup your brokers properly (
https://kafka.apache.org/090/documentation.html#security_overview)
2. You have to manually create the client JAAS file (
https://kafka.apache.org/090/documentation.html#security_overview)
   a sample file would look like this:
     KafkaClient {

        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        keyTab="/etc/security/keytabs/kafka_client.keytab"
        principal="kafka-client-1@EXAMPLE.COM";
    };

3. On the operator you have to set the attribute JVM_OPTS
 -Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf

4. On the operator you have to set the consumerProperties, for example

set dt.operator.$your_operator_name.{consumerProps.security.protocol}  to
 SASL or other security function you use
set dt.operator.$your_operator_name.{consumerPrope.
sasl.kerberos.service.name} to kafka


Hope this would help you!


Regards,
Siyuan



On Sun, Oct 30, 2016 at 10:56 PM, Raja.Aravapalli <
Raja.Aravapalli@target.com> wrote:

>
>
> Hi Team,
>
>
>
> Can someone pls help me with below requested information ?
>
>
>
> Does apache apex have any inbuilt kafka input operator to read from Kafka
> 0.9 secure kafka topics?
>
>
>
> Thanks a lot.
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *"Raja.Aravapalli" <Ra...@target.com>
> *Reply-To: *"users@apex.apache.org" <us...@apex.apache.org>
> *Date: *Monday, October 24, 2016 at 2:29 PM
> *To: *"users@apex.apache.org" <us...@apex.apache.org>
> *Subject: *[EXTERNAL] kafka
>
>
>
>
>
> Hi,
>
>
>
> Do we have any kaka operator readily available to consume messages from
> secure kafka topics in kafka 0.9 clusters?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>

Re: [EXTERNAL] kafka

Posted by "Raja.Aravapalli" <Ra...@target.com>.
Hi Team,

Can someone pls help me with below requested information ?

Does apache apex have any inbuilt kafka input operator to read from Kafka 0.9 secure kafka topics?

Thanks a lot.

Regards,
Raja.

From: "Raja.Aravapalli" <Ra...@target.com>
Reply-To: "users@apex.apache.org" <us...@apex.apache.org>
Date: Monday, October 24, 2016 at 2:29 PM
To: "users@apex.apache.org" <us...@apex.apache.org>
Subject: [EXTERNAL] kafka


Hi,

Do we have any kaka operator readily available to consume messages from secure kafka topics in kafka 0.9 clusters?

Thanks a lot.


Regards,
Raja.

Re: kafka

Posted by Jaspal Singh <ja...@gmail.com>.
There are couple of Kafka input operators like
KafkaSinglePortBytesInputOperator and KafkaSinglePortInputOperator that
extends AbstractKafkaInputOperator. They are part of malhar library.

They can right away consume messages from kafka topics.

Thanks!!

On Monday, October 24, 2016, Raja.Aravapalli <Ra...@target.com>
wrote:

>
>
> Hi,
>
>
>
> Do we have any kaka operator readily available to consume messages from
> secure kafka topics in kafka 0.9 clusters?
>
>
>
> Thanks a lot.
>
>
>
>
>
> Regards,
>
> Raja.
>