You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishwas Siravara <vs...@gmail.com> on 2019/09/03 02:04:29 UTC
understanding task manager logs
Hi guys,
I am using flink 1.7.2 and my application consumes from a kafka topic and
publish to another kafka topic which is in its own kafka environment
running a different kafka version,. I am using FlinkKafkaConsumer010 from
this dependency
*"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. *
In the task manager log I see these lines:
2019-09-02 02:57:59,840 INFO
org.apache.kafka.common.security.authenticator.AbstractLogin -
Successfully logged in.
2019-09-02 02:57:59,841 INFO
org.apache.kafka.common.security.kerberos.KerberosLogin -
[Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT refresh
thread started.
2019-09-02 02:57:59,842 INFO
org.apache.kafka.common.security.kerberos.KerberosLogin -
[Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT valid
starting at: Mon Sep 02 02:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO
org.apache.kafka.common.security.kerberos.KerberosLogin -
[Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT
expires: Mon Sep 02 12:57:59 GMT 2019
2019-09-02 02:57:59,843 INFO
org.apache.kafka.common.security.kerberos.KerberosLogin -
[Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT refresh
sleeping until: Mon Sep 02 11:14:13 GMT 2019
2019-09-02 02:57:59,919 WARN
org.apache.kafka.clients.consumer.ConsumerConfig - The
configuration 'zookeeper.connect' was supplied but isn't a known
config.*2019-09-02 02:57:59,919 INFO
org.apache.kafka.common.utils.AppInfoParser - Kafka
version : 0.10.2.0
*2019-09-02 02:57:59,919 INFO
org.apache.kafka.common.utils.AppInfoParser - Kafka
commitId : 576d93a8dc0cf421
Here if you see the Kafka version is 0.10.2.0. Is this the version the
broker is running or is this coming from flink ? I have forced the
kafka-client version
to be 2.2.0
"org.apache.kafka" % "kafka-clients" % "2.2.0" force()
I also don't see 0.10.2.0 in the dependency tree of my build.
Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ?
What should I do if the consumer broker and producer broker are on
different versions of kafka ?
Thanks,
Vishwas
Thanks,
Vishwas
Re: understanding task manager logs
Posted by Vishwas Siravara <vs...@gmail.com>.
Thanks, I'll check it out.
On Thu, Sep 5, 2019 at 5:22 AM Fabian Hueske <fh...@gmail.com> wrote:
> Hi Vishwas,
>
> This is a log statement from Kafka [1].
> Not sure how when AppInfoParser is created (the log message is written by
> the constructor).
>
> For Kafka versions > 1.0, I'd recommend the universal connector [2].
>
> Not sure how well it works if producers and consumers have different
> versions.
> Maybe Gordon (in CC) has some experience with that.
>
> Best, Fabian
>
> [1]
> https://github.com/axbaretto/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/AppInfoParser.java#L117
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kafka.html#kafka-100-connector
>
> Am Di., 3. Sept. 2019 um 04:04 Uhr schrieb Vishwas Siravara <
> vsiravara@gmail.com>:
>
>> Hi guys,
>> I am using flink 1.7.2 and my application consumes from a kafka topic and
>> publish to another kafka topic which is in its own kafka environment
>> running a different kafka version,. I am using FlinkKafkaConsumer010 from
>> this dependency
>> *"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. *
>>
>> In the task manager log I see these lines:
>>
>> 2019-09-02 02:57:59,840 INFO org.apache.kafka.common.security.authenticator.AbstractLogin - Successfully logged in.
>> 2019-09-02 02:57:59,841 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT refresh thread started.
>> 2019-09-02 02:57:59,842 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT valid starting at: Mon Sep 02 02:57:59 GMT 2019
>> 2019-09-02 02:57:59,843 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT expires: Mon Sep 02 12:57:59 GMT 2019
>> 2019-09-02 02:57:59,843 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT refresh sleeping until: Mon Sep 02 11:14:13 GMT 2019
>> 2019-09-02 02:57:59,919 WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.*2019-09-02 02:57:59,919 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.10.2.0
>> *2019-09-02 02:57:59,919 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 576d93a8dc0cf421
>>
>> Here if you see the Kafka version is 0.10.2.0. Is this the version the broker is running or is this coming from flink ? I have forced the kafka-client version
>>
>> to be 2.2.0
>>
>> "org.apache.kafka" % "kafka-clients" % "2.2.0" force()
>>
>> I also don't see 0.10.2.0 in the dependency tree of my build.
>>
>> Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ? What should I do if the consumer broker and producer broker are on different versions of kafka ?
>>
>>
>> Thanks,
>>
>> Vishwas
>>
>>
>> Thanks,
>>
>> Vishwas
>>
>>
>>
>>
Re: understanding task manager logs
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Vishwas,
This is a log statement from Kafka [1].
Not sure how when AppInfoParser is created (the log message is written by
the constructor).
For Kafka versions > 1.0, I'd recommend the universal connector [2].
Not sure how well it works if producers and consumers have different
versions.
Maybe Gordon (in CC) has some experience with that.
Best, Fabian
[1]
https://github.com/axbaretto/kafka/blob/master/clients/src/main/java/org/apache/kafka/common/utils/AppInfoParser.java#L117
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kafka.html#kafka-100-connector
Am Di., 3. Sept. 2019 um 04:04 Uhr schrieb Vishwas Siravara <
vsiravara@gmail.com>:
> Hi guys,
> I am using flink 1.7.2 and my application consumes from a kafka topic and
> publish to another kafka topic which is in its own kafka environment
> running a different kafka version,. I am using FlinkKafkaConsumer010 from
> this dependency
> *"org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion. *
>
> In the task manager log I see these lines:
>
> 2019-09-02 02:57:59,840 INFO org.apache.kafka.common.security.authenticator.AbstractLogin - Successfully logged in.
> 2019-09-02 02:57:59,841 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT refresh thread started.
> 2019-09-02 02:57:59,842 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT valid starting at: Mon Sep 02 02:57:59 GMT 2019
> 2019-09-02 02:57:59,843 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT expires: Mon Sep 02 12:57:59 GMT 2019
> 2019-09-02 02:57:59,843 INFO org.apache.kafka.common.security.kerberos.KerberosLogin - [Principal=kafka/sl73rspapd035.visa.com@CORPDEV.VISA.COM]: TGT refresh sleeping until: Mon Sep 02 11:14:13 GMT 2019
> 2019-09-02 02:57:59,919 WARN org.apache.kafka.clients.consumer.ConsumerConfig - The configuration 'zookeeper.connect' was supplied but isn't a known config.*2019-09-02 02:57:59,919 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version : 0.10.2.0
> *2019-09-02 02:57:59,919 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId : 576d93a8dc0cf421
>
> Here if you see the Kafka version is 0.10.2.0. Is this the version the broker is running or is this coming from flink ? I have forced the kafka-client version
>
> to be 2.2.0
>
> "org.apache.kafka" % "kafka-clients" % "2.2.0" force()
>
> I also don't see 0.10.2.0 in the dependency tree of my build.
>
> Also will flink-connector-kafka-0.10 work for kafka versions > 1.0 ? What should I do if the consumer broker and producer broker are on different versions of kafka ?
>
>
> Thanks,
>
> Vishwas
>
>
> Thanks,
>
> Vishwas
>
>
>
>