You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Harikiran Nayak <ha...@streamsets.com> on 2016/03/03 17:18:25 UTC

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Michal,

Are you able to write and read from the kerberized Kafka setup using the
Kafka Console Producer and Consumer?

I am taking a look at your configuration files.

Thanks
Hari.

On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com>
wrote:

> Hey Michal,
>
> I'm cc'ing the StreamSets user list, which might be able to get you some
> better StreamSets-specific answers.
>
> Thanks!
> Natty
>
> On Thursday, March 3, 2016, Michał Kabocik <mi...@gmail.com>
> wrote:
>
>> Dears,
>>
>> I’m Middleware Engineer and I’m trying to configure secure Kafka Cluster
>> with SSL and Kerberos authentication with StreamSets, which will be used
>> for data injection to HDP.
>>
>> I have two Kafka Clusters; one with SSL enabled and there I successfully
>> connected StreamSets to Kafka with SSL authentication, and second one with
>> Kerberos authentication and here I’m facing with the problem:
>>
>> Both Kafka (with Zookeeper) and StreamSets are configured to authenticate
>> via Kerberos. When starting all of them, I see in the logs, that they are
>> successfully authenticated (TGT granted etc.)
>>
>> I have two listeners defined in Kafka:
>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I
>> see Kafka listens on both, 9092 and 9093.
>>
>> When I connect StreamSets to Kafka on port 9092, everything works smooth.
>> But when I try to connect to port 9093, error occurs:
>>
>> KAFKA_41 - Could not get partition count for topic 'streamsets5' :
>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get
>> caseition count for topic 'streamsets5' :
>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>
>> I see no errors in Kafka, in the log of StreamSets, there is only above
>> error visible. I attached major config files of Kafka, Zookeeper and
>> StreamSets.
>>
>> Will greatly appreciate your help in solving this case!
>>
>> Kind regards,
>>
>
>
> --
> Jonathan "Natty" Natkins
> StreamSets | Field Engineer
> mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "sdc-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sdc-user+unsubscribe@streamsets.com.
> Visit this group at
> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Posted by Michał Kabocik <mi...@gmail.com>.
Dear Hari,

Thank you for your reply.

Replying to your questions:
Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
I followed your suggestion and added mentioned entries in 
server.properties_krb5. Now when starting Kafka Broker I see:
listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
sasl.kerberos.service.name = kafka
advertised.host.name = plx164h.xx.xxx.xx

Unfortunately it didn't help. Error in StreamSets is the same. I've tried 
to use built-in kafka console consumer and also not succeded. Here is my 
config:

On host A I have Kafka broker which is running with the config from 
previous email. On host B, I have another Kafka from which I used console 
consumer with following config:

kafka_client_jaas.conf:
KafkaClient {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        keyTab="/etc/security/keytabs/kafka_client.service.keytab"
        principal="client/10.xxx.xxx.72@HDPCYBERSECACC.XX.XX";
    };

consumer.properties:
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=client

I'm starting console consumer with the command:
./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic 
streamsets2 --new-consumer --consumer.config consumer.properties

When started, there is no error, console consumer seems to work fine, but 
when producing to this topic, no messages are read. 
>From kerberos side everything looks correct:

Mar 04 16:00:31 lxhnl219.srv.pl.ing.net krb5kdc[16307](info): AS_REQ (4 
etypes {18 17 16 23}) 10.111.159.72: ISSUE: authtime 1457103631, etypes 
{rep=18 tkt=18 ses=18}, client/10.xxx.xxxx.72@HDPCYBERSECACC.XX.XX for 
krbtgt/HDPCYBERSECACC.XX.XX@HDPCYBERSECACC.XX.XX
Mar 04 16:00:31 lxhnl219.srv.pl.ing.net krb5kdc[16307](info): TGS_REQ (4 
etypes {18 17 16 23}) 10.111.159.72: ISSUE: authtime 1457103631, etypes 
{rep=18 tkt=18 ses=18}, client/10.xxx.xxx.72@HDPCYBERSECACC.XX.XX for 
client/plx164h.XX.XX@HDPCYBERSECACC.XX.XX

Could you please a look at this? Maybe you see configuration error?

Kind regards,
Michal


W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak 
napisał:

> Hi Michal,
>
> Can you please add the *advertised.listeners* and *advertised.host.name 
> <http://advertised.host.name>* properties in your kafka server config 
> file 'server.properties_krb5'?
>
> For example, I have the following configuration in my working setup
>
> listeners=SASL_PLAINTEXT://:9092
> advertised.listeners=SASL_PLAINTEXT://:9092
> host.name=kafka
> advertised.host.name=kafka
>
> 'kafka' is the hostname on which the Kafka broker is running in my setup. 
> There is an entry for this host in '/etc/hosts' on the node where 
> StreamSets is running.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com 
> <javascript:>> wrote:
>
>> Hi Michal,
>>
>> Are you able to write and read from the kerberized Kafka setup using the 
>> Kafka Console Producer and Consumer?
>>
>> I am taking a look at your configuration files.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com 
>> <javascript:>> wrote:
>>
>>> Hey Michal,
>>>
>>> I'm cc'ing the StreamSets user list, which might be able to get you 
>>> some better StreamSets-specific answers.
>>>
>>> Thanks!
>>> Natty
>>>
>>> On Thursday, March 3, 2016, Michał Kabocik <michal....@gmail.com 
>>> <javascript:>> wrote:
>>>
>>>> Dears,
>>>>
>>>> I’m Middleware Engineer and I’m trying to configure secure Kafka 
>>>> Cluster with SSL and Kerberos authentication with StreamSets, which will be 
>>>> used for data injection to HDP.
>>>>
>>>> I have two Kafka Clusters; one with SSL enabled and there I 
>>>> successfully connected StreamSets to Kafka with SSL authentication, and 
>>>> second one with Kerberos authentication and here I’m facing with the 
>>>> problem:
>>>>
>>>> Both Kafka (with Zookeeper) and StreamSets are configured to 
>>>> authenticate via Kerberos. When starting all of them, I see in the logs, 
>>>> that they are successfully authenticated (TGT granted etc.)
>>>>
>>>> I have two listeners defined in Kafka: 
>>>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I 
>>>> see Kafka listens on both, 9092 and 9093.
>>>>
>>>> When I connect StreamSets to Kafka on port 9092, everything works 
>>>> smooth. But when I try to connect to port 9093, error occurs:
>>>>
>>>> KAFKA_41 - Could not get partition count for topic 'streamsets5' : 
>>>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get 
>>>> caseition count for topic 'streamsets5' : 
>>>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>>>
>>>> I see no errors in Kafka, in the log of StreamSets, there is only above 
>>>> error visible. I attached major config files of Kafka, Zookeeper and 
>>>> StreamSets.
>>>>
>>>> Will greatly appreciate your help in solving this case!
>>>>
>>>> Kind regards,
>>>>
>>>
>>>
>>> -- 
>>> Jonathan "Natty" Natkins
>>> StreamSets | Field Engineer
>>> mobile: 609.577.1600 <#> | linkedin 
>>> <http://www.linkedin.com/in/nattyice>
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "sdc-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to sdc-user+u...@streamsets.com <javascript:>.
>>> Visit this group at 
>>> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>>>
>>

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Posted by Harikiran Nayak <ha...@streamsets.com>.
Hi Michal,

The configuration in consumer.properties is not correct.
The 'sasl.kerberos.service.name' option expects the kerberos principal that
Kafka runs as.
In your case it should be '*sasl.kerberos.service.name
<http://sasl.kerberos.service.name/>=*kafka*' *

Can you please test using the Kafka Console Producer as well?
This will make sure that your kerberos setup is right.

Secondly, you must specify the same 'security.protocol' and '
sasl.kerberos.service.name' options in the Kafka Consumer stage in
StreamSets pipeline.
See attached snapshot that shows how to specify these properties.

Thanks
Hari.



*[image: Screen Shot 2016-03-04 at 10.32.42 AM.png]*

On Fri, Mar 4, 2016 at 7:16 AM Michał Kabocik <mi...@gmail.com>
wrote:

> Dear Hari,
>
> Thank you for your reply.
>
> Replying to your questions:
> Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
> I followed your suggestion and added mentioned entries in
> server.properties_krb5. Now when starting Kafka Broker I see:
> listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
> advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
> sasl.kerberos.service.name = kafka
> advertised.host.name = plx164h.xx.xxx.xx
>
> Unfortunately it didn't help. Error in StreamSets is the same. I've tried
> to use built-in kafka console consumer and also not succeded. Here is my
> config:
>
> On host A I have Kafka broker which is running with the config from
> previous email. On host B, I have another Kafka from which I used console
> consumer with following config:
>
> kafka_client_jaas.conf:
> KafkaClient {
>         com.sun.security.auth.module.Krb5LoginModule required
>         useKeyTab=true
>         storeKey=true
>         keyTab="/etc/security/keytabs/kafka_client.service.keytab"
>         principal="client/10.xxx.xxx.72@HDPCYBERSECACC.XX.XX";
>     };
>
> consumer.properties:
> security.protocol=SASL_PLAINTEXT
> sasl.kerberos.service.name=client
>
> I'm starting console consumer with the command:
> ./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic
> streamsets2 --new-consumer --consumer.config consumer.properties
>
> When started, there is no error, console consumer seems to work fine, but
> when producing to this topic, no messages are read.
> From kerberos side everything looks correct:
>
> Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): AS_REQ (4 etypes {18
> 17 16 23}) 10.xxx.xxx.72
> <http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>:
> ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
> client/10.xxx.xxxx.72@HDPCYBERSECACC.XX.XX for
> krbtgt/HDPCYBERSECACC.XX.XX@HDPCYBERSECACC.XX.XX
> Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): TGS_REQ (4 etypes {18
> 17 16 23}) 10.xxx.xxx.72
> <http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>:
> ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
> client/10.xxx.xxx.72@HDPCYBERSECACC.XX.XX for
> client/plx164h.XX.XX@HDPCYBERSECACC.XX.XX
>
> Could you please a look at this? Maybe you see configuration error?
>
> Kind regards,
> Michal
>
> W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak
> napisał:
>
>> Hi Michal,
>>
>> Can you please add the *advertised.listeners* and *advertised.host.name
>> <http://advertised.host.name>* properties in your kafka server config
>> file 'server.properties_krb5'?
>>
>> For example, I have the following configuration in my working setup
>>
>> listeners=SASL_PLAINTEXT://:9092
>> advertised.listeners=SASL_PLAINTEXT://:9092
>> host.name=kafka
>> advertised.host.name=kafka
>>
>> 'kafka' is the hostname on which the Kafka broker is running in my setup.
>> There is an entry for this host in '/etc/hosts' on the node where
>> StreamSets is running.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com>
>> wrote:
>>
> Hi Michal,
>>>
>>> Are you able to write and read from the kerberized Kafka setup using the
>>> Kafka Console Producer and Consumer?
>>>
>>> I am taking a look at your configuration files.
>>>
>>> Thanks
>>> Hari.
>>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com>
>>> wrote:
>>>
>> Hey Michal,
>>>>
>>>> I'm cc'ing the StreamSets user list, which might be able to get you
>>>> some better StreamSets-specific answers.
>>>>
>>>> Thanks!
>>>>
>>> Natty
>>>>
>>>
>>>>
>>>> On Thursday, March 3, 2016, Michał Kabocik <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Dears,
>>>>>
>>>>> I’m Middleware Engineer and I’m trying to configure secure Kafka
>>>>> Cluster with SSL and Kerberos authentication with StreamSets, which will be
>>>>> used for data injection to HDP.
>>>>>
>>>>> I have two Kafka Clusters; one with SSL enabled and there I
>>>>> successfully connected StreamSets to Kafka with SSL authentication, and
>>>>> second one with Kerberos authentication and here I’m facing with the
>>>>> problem:
>>>>>
>>>>> Both Kafka (with Zookeeper) and StreamSets are configured to
>>>>> authenticate via Kerberos. When starting all of them, I see in the logs,
>>>>> that they are successfully authenticated (TGT granted etc.)
>>>>>
>>>>> I have two listeners defined in Kafka:
>>>>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I
>>>>> see Kafka listens on both, 9092 and 9093.
>>>>>
>>>>> When I connect StreamSets to Kafka on port 9092, everything works
>>>>> smooth. But when I try to connect to port 9093, error occurs:
>>>>>
>>>>> KAFKA_41 - Could not get partition count for topic 'streamsets5' :
>>>>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get
>>>>> caseition count for topic 'streamsets5' :
>>>>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>>>>
>>>>> I see no errors in Kafka, in the log of StreamSets, there is only
>>>>> above error visible. I attached major config files of Kafka, Zookeeper and
>>>>> StreamSets.
>>>>>
>>>>> Will greatly appreciate your help in solving this case!
>>>>>
>>>>> Kind regards,
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>> Jonathan "Natty" Natkins
>>>> StreamSets | Field Engineer
>>>>
>>> mobile: 609.577.1600[image: Auto Generated Inline Image 1]
>>>> <#msg-f:1527884890376799684_> | linkedin
>>>> <http://www.linkedin.com/in/nattyice>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "sdc-user" group.
>>>>
>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to sdc-user+u...@streamsets.com.
>>>
>>>
>>>> Visit this group at
>>>> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "sdc-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sdc-user+unsubscribe@streamsets.com.
> Visit this group at
> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Posted by Michał Kabocik <mi...@gmail.com>.
Dear Hari,

Thank you for your reply.

Replying to your questions:
Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
I followed your suggestion and added mentioned entries in 
server.properties_krb5. Now when starting Kafka Broker I see:
listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
sasl.kerberos.service.name = kafka
advertised.host.name = plx164h.xx.xxx.xx

Unfortunately it didn't help. Error in StreamSets is the same. I've tried 
to use built-in kafka console consumer and also not succeded. Here is my 
config:

On host A I have Kafka broker which is running with the config from 
previous email. On host B, I have another Kafka from which I used console 
consumer with following config:

kafka_client_jaas.conf:
KafkaClient {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        keyTab="/etc/security/keytabs/kafka_client.service.keytab"
        principal="client/10.xxx.xxx.72@HDPCYBERSECACC.XX.XX";
    };

consumer.properties:
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=client

I'm starting console consumer with the command:
./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic 
streamsets2 --new-consumer --consumer.config consumer.properties

When started, there is no error, console consumer seems to work fine, but 
when producing to this topic, no messages are read. 
>From kerberos side everything looks correct:

Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): AS_REQ (4 etypes {18 
17 16 23}) 10.xxx.xxx.72 
<http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>: 
ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18}, 
client/10.xxx.xxxx.72@HDPCYBERSECACC.XX.XX for 
krbtgt/HDPCYBERSECACC.XX.XX@HDPCYBERSECACC.XX.XX
Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): TGS_REQ (4 etypes {18 
17 16 23}) 10.xxx.xxx.72 
<http://www.google.com/url?q=http%3A%2F%2F10.111.159.72&sa=D&sntz=1&usg=AFQjCNGfeOUI5N_QC-VKyS_d9ouhNTfYpQ>: 
ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18}, 
client/10.xxx.xxx.72@HDPCYBERSECACC.XX.XX for 
client/plx164h.XX.XX@HDPCYBERSECACC.XX.XX

Could you please a look at this? Maybe you see configuration error?

Kind regards,
Michal

W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak 
napisał:

> Hi Michal,
>
> Can you please add the *advertised.listeners* and *advertised.host.name 
> <http://advertised.host.name>* properties in your kafka server config 
> file 'server.properties_krb5'?
>
> For example, I have the following configuration in my working setup
>
> listeners=SASL_PLAINTEXT://:9092
> advertised.listeners=SASL_PLAINTEXT://:9092
> host.name=kafka
> advertised.host.name=kafka
>
> 'kafka' is the hostname on which the Kafka broker is running in my setup. 
> There is an entry for this host in '/etc/hosts' on the node where 
> StreamSets is running.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com 
> <javascript:>> wrote:
>
>> Hi Michal,
>>
>> Are you able to write and read from the kerberized Kafka setup using the 
>> Kafka Console Producer and Consumer?
>>
>> I am taking a look at your configuration files.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com 
>> <javascript:>> wrote:
>>
>>> Hey Michal,
>>>
>>> I'm cc'ing the StreamSets user list, which might be able to get you 
>>> some better StreamSets-specific answers.
>>>
>>> Thanks!
>>> Natty
>>>
>>> On Thursday, March 3, 2016, Michał Kabocik <michal....@gmail.com 
>>> <javascript:>> wrote:
>>>
>>>> Dears,
>>>>
>>>> I’m Middleware Engineer and I’m trying to configure secure Kafka 
>>>> Cluster with SSL and Kerberos authentication with StreamSets, which will be 
>>>> used for data injection to HDP.
>>>>
>>>> I have two Kafka Clusters; one with SSL enabled and there I 
>>>> successfully connected StreamSets to Kafka with SSL authentication, and 
>>>> second one with Kerberos authentication and here I’m facing with the 
>>>> problem:
>>>>
>>>> Both Kafka (with Zookeeper) and StreamSets are configured to 
>>>> authenticate via Kerberos. When starting all of them, I see in the logs, 
>>>> that they are successfully authenticated (TGT granted etc.)
>>>>
>>>> I have two listeners defined in Kafka: 
>>>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I 
>>>> see Kafka listens on both, 9092 and 9093.
>>>>
>>>> When I connect StreamSets to Kafka on port 9092, everything works 
>>>> smooth. But when I try to connect to port 9093, error occurs:
>>>>
>>>> KAFKA_41 - Could not get partition count for topic 'streamsets5' : 
>>>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get 
>>>> caseition count for topic 'streamsets5' : 
>>>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>>>
>>>> I see no errors in Kafka, in the log of StreamSets, there is only above 
>>>> error visible. I attached major config files of Kafka, Zookeeper and 
>>>> StreamSets.
>>>>
>>>> Will greatly appreciate your help in solving this case!
>>>>
>>>> Kind regards,
>>>>
>>>
>>>
>>> -- 
>>> Jonathan "Natty" Natkins
>>> StreamSets | Field Engineer
>>> mobile: 609.577.1600 <#> | linkedin 
>>> <http://www.linkedin.com/in/nattyice>
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "sdc-user" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to sdc-user+u...@streamsets.com <javascript:>.
>>> Visit this group at 
>>> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>>>
>>

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Posted by Ismael Juma <is...@juma.me.uk>.
Hi Harikiran,

One comment: `advertised.host.name` is not used if `advertised.listeners`
is set and similarly `host.name` is not used if `listeners` is set. In
general, the use of those properties is now discouraged in favour of
listeners. There is a PR to make the documentation clearer:

https://github.com/apache/kafka/pull/793

Ismael

On Thu, Mar 3, 2016 at 4:48 PM, Harikiran Nayak <ha...@streamsets.com> wrote:

> Hi Michal,
>
> Can you please add the *advertised.listeners* and *advertised.host.name
> <http://advertised.host.name>* properties in your kafka server config file
> 'server.properties_krb5'?
>
> For example, I have the following configuration in my working setup
>
> listeners=SASL_PLAINTEXT://:9092
> advertised.listeners=SASL_PLAINTEXT://:9092
> host.name=kafka
> advertised.host.name=kafka
>
> 'kafka' is the hostname on which the Kafka broker is running in my setup.
> There is an entry for this host in '/etc/hosts' on the node where
> StreamSets is running.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com>
> wrote:
>
> > Hi Michal,
> >
> > Are you able to write and read from the kerberized Kafka setup using the
> > Kafka Console Producer and Consumer?
> >
> > I am taking a look at your configuration files.
> >
> > Thanks
> > Hari.
> >
> > On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com>
> > wrote:
> >
> >> Hey Michal,
> >>
> >> I'm cc'ing the StreamSets user list, which might be able to get you some
> >> better StreamSets-specific answers.
> >>
> >> Thanks!
> >> Natty
> >>
> >> On Thursday, March 3, 2016, Michał Kabocik <mi...@gmail.com>
> >> wrote:
> >>
> >>> Dears,
> >>>
> >>> I’m Middleware Engineer and I’m trying to configure secure Kafka
> Cluster
> >>> with SSL and Kerberos authentication with StreamSets, which will be
> used
> >>> for data injection to HDP.
> >>>
> >>> I have two Kafka Clusters; one with SSL enabled and there I
> successfully
> >>> connected StreamSets to Kafka with SSL authentication, and second one
> with
> >>> Kerberos authentication and here I’m facing with the problem:
> >>>
> >>> Both Kafka (with Zookeeper) and StreamSets are configured to
> >>> authenticate via Kerberos. When starting all of them, I see in the
> logs,
> >>> that they are successfully authenticated (TGT granted etc.)
> >>>
> >>> I have two listeners defined in Kafka:
> >>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting
> Kafka, I
> >>> see Kafka listens on both, 9092 and 9093.
> >>>
> >>> When I connect StreamSets to Kafka on port 9092, everything works
> >>> smooth. But when I try to connect to port 9093, error occurs:
> >>>
> >>> KAFKA_41 - Could not get partition count for topic 'streamsets5' :
> >>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get
> >>> caseition count for topic 'streamsets5' :
> >>> org.apache.kafka.common.KafkaException: Failed to construct kafka
> consumer
> >>>
> >>> I see no errors in Kafka, in the log of StreamSets, there is only above
> >>> error visible. I attached major config files of Kafka, Zookeeper and
> >>> StreamSets.
> >>>
> >>> Will greatly appreciate your help in solving this case!
> >>>
> >>> Kind regards,
> >>>
> >>
> >>
> >> --
> >> Jonathan "Natty" Natkins
> >> StreamSets | Field Engineer
> >> mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice>
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "sdc-user" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to sdc-user+unsubscribe@streamsets.com.
> >> Visit this group at
> >> https://groups.google.com/a/streamsets.com/group/sdc-user/.
> >>
> >
>

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Posted by Harikiran Nayak <ha...@streamsets.com>.
Hi Michal,

Can you please add the *advertised.listeners* and *advertised.host.name
<http://advertised.host.name>* properties in your kafka server config file
'server.properties_krb5'?

For example, I have the following configuration in my working setup

listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://:9092
host.name=kafka
advertised.host.name=kafka

'kafka' is the hostname on which the Kafka broker is running in my setup.
There is an entry for this host in '/etc/hosts' on the node where
StreamSets is running.

Thanks
Hari.

On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak <ha...@streamsets.com> wrote:

> Hi Michal,
>
> Are you able to write and read from the kerberized Kafka setup using the
> Kafka Console Producer and Consumer?
>
> I am taking a look at your configuration files.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins <na...@streamsets.com>
> wrote:
>
>> Hey Michal,
>>
>> I'm cc'ing the StreamSets user list, which might be able to get you some
>> better StreamSets-specific answers.
>>
>> Thanks!
>> Natty
>>
>> On Thursday, March 3, 2016, Michał Kabocik <mi...@gmail.com>
>> wrote:
>>
>>> Dears,
>>>
>>> I’m Middleware Engineer and I’m trying to configure secure Kafka Cluster
>>> with SSL and Kerberos authentication with StreamSets, which will be used
>>> for data injection to HDP.
>>>
>>> I have two Kafka Clusters; one with SSL enabled and there I successfully
>>> connected StreamSets to Kafka with SSL authentication, and second one with
>>> Kerberos authentication and here I’m facing with the problem:
>>>
>>> Both Kafka (with Zookeeper) and StreamSets are configured to
>>> authenticate via Kerberos. When starting all of them, I see in the logs,
>>> that they are successfully authenticated (TGT granted etc.)
>>>
>>> I have two listeners defined in Kafka:
>>> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I
>>> see Kafka listens on both, 9092 and 9093.
>>>
>>> When I connect StreamSets to Kafka on port 9092, everything works
>>> smooth. But when I try to connect to port 9093, error occurs:
>>>
>>> KAFKA_41 - Could not get partition count for topic 'streamsets5' :
>>> com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get
>>> caseition count for topic 'streamsets5' :
>>> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>>>
>>> I see no errors in Kafka, in the log of StreamSets, there is only above
>>> error visible. I attached major config files of Kafka, Zookeeper and
>>> StreamSets.
>>>
>>> Will greatly appreciate your help in solving this case!
>>>
>>> Kind regards,
>>>
>>
>>
>> --
>> Jonathan "Natty" Natkins
>> StreamSets | Field Engineer
>> mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "sdc-user" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to sdc-user+unsubscribe@streamsets.com.
>> Visit this group at
>> https://groups.google.com/a/streamsets.com/group/sdc-user/.
>>
>