You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by sandeep6 <vr...@gmail.com> on 2017/04/18 21:00:48 UTC
Re: Flink Kafka Consumer Behaviour
Is this fixed now? If not, is there any way to monitor kafka offset that is
being processed by Flink? This should be a use case for everyone who uses
Flink with Kafka.
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-tp8257p12663.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink Kafka Consumer Behaviour
Posted by Stephan Ewen <se...@apache.org>.
The latest Flink has consumers for Kafka 0.8, 0.9, 0.10 - which one are you
using?
I would assume you use Flink with Kafka 0.8.x, because as far as I know,
starting from Kafka 0.9, offsets are not handled by ZooKeeper any more...
On Mon, Apr 24, 2017 at 12:18 AM, Meghashyam Sandeep V <
vr1meghashyam@gmail.com> wrote:
> Hi All,
>
> Sorry for the miscommunication. I'm not using 0.8. I'm using latest
> available flink-kafka client. I don't see my app registered as a consumer
> group. I wanted to know if there is a way to monitor Kafka offsets.
>
> Thanks,
> Sandeep
>
> On Apr 23, 2017 9:38 AM, "Stephan Ewen" <se...@apache.org> wrote:
>
>> Since it is something special to Kafka 0.8, it could be implemented in a
>> simple addition to the ZooKeeperOffsetHandler used by the
>> FlinkKafkaConsumer08.
>>
>> Would you be willing to contribute this? That would certainly help
>> speeding up the resolution of the issue...
>>
>>
>> On Fri, Apr 21, 2017 at 2:33 AM, Tzu-Li (Gordon) Tai <tzulitai@apache.org
>> > wrote:
>>
>>> One additional note:
>>>
>>> In FlinkKafkaConsumer 0.9+, the current read offset should already exist
>>> in Flink metrics.
>>> See https://issues.apache.org/jira/browse/FLINK-4186.
>>>
>>> But yes, this is still missing for 0.8, so you need to directly query ZK
>>> for this.
>>>
>>> Cheers,
>>> Gordon
>>>
>>>
>>> On 21 April 2017 at 8:28:09 AM, Tzu-Li (Gordon) Tai (tzulitai@apache.org)
>>> wrote:
>>>
>>> Hi Sandeep,
>>>
>>> It isn’t fixed yet, so I think external tools like the Kafka offset
>>> checker still won’t work.
>>> If you’re using 08 and is currently stuck with this issue, you can still
>>> directly query ZK to get the offsets.
>>>
>>> I think for FlinkKafkaConsumer09 the offset is exposed to Flink's metric
>>> system using Kafka’s own returned metrics, but for 08 this is still missing.
>>>
>>> There is this JIRA [1] that aims at exposing consumer lag across all
>>> Kafka consumer versions to Flink metrics. Perhaps it would make sense to
>>> also generally expose the offset for all Kafka consumer versions to Flink
>>> metrics as well.
>>>
>>> - Gordon
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-6109
>>>
>>>
>>> On 19 April 2017 at 5:11:11 AM, sandeep6 (vr1meghashyam@gmail.com)
>>> wrote:
>>>
>>> Is this fixed now? If not, is there any way to monitor kafka offset that
>>> is
>>> being processed by Flink? This should be a use case for everyone who uses
>>> Flink with Kafka.
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-flink-user-maili
>>> ng-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-
>>> Behaviour-tp8257p12663.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>>
>>
Re: Flink Kafka Consumer Behaviour
Posted by Meghashyam Sandeep V <vr...@gmail.com>.
Hi All,
Sorry for the miscommunication. I'm not using 0.8. I'm using latest
available flink-kafka client. I don't see my app registered as a consumer
group. I wanted to know if there is a way to monitor Kafka offsets.
Thanks,
Sandeep
On Apr 23, 2017 9:38 AM, "Stephan Ewen" <se...@apache.org> wrote:
> Since it is something special to Kafka 0.8, it could be implemented in a
> simple addition to the ZooKeeperOffsetHandler used by the
> FlinkKafkaConsumer08.
>
> Would you be willing to contribute this? That would certainly help
> speeding up the resolution of the issue...
>
>
> On Fri, Apr 21, 2017 at 2:33 AM, Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> One additional note:
>>
>> In FlinkKafkaConsumer 0.9+, the current read offset should already exist
>> in Flink metrics.
>> See https://issues.apache.org/jira/browse/FLINK-4186.
>>
>> But yes, this is still missing for 0.8, so you need to directly query ZK
>> for this.
>>
>> Cheers,
>> Gordon
>>
>>
>> On 21 April 2017 at 8:28:09 AM, Tzu-Li (Gordon) Tai (tzulitai@apache.org)
>> wrote:
>>
>> Hi Sandeep,
>>
>> It isn’t fixed yet, so I think external tools like the Kafka offset
>> checker still won’t work.
>> If you’re using 08 and is currently stuck with this issue, you can still
>> directly query ZK to get the offsets.
>>
>> I think for FlinkKafkaConsumer09 the offset is exposed to Flink's metric
>> system using Kafka’s own returned metrics, but for 08 this is still missing.
>>
>> There is this JIRA [1] that aims at exposing consumer lag across all
>> Kafka consumer versions to Flink metrics. Perhaps it would make sense to
>> also generally expose the offset for all Kafka consumer versions to Flink
>> metrics as well.
>>
>> - Gordon
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-6109
>>
>>
>> On 19 April 2017 at 5:11:11 AM, sandeep6 (vr1meghashyam@gmail.com) wrote:
>>
>> Is this fixed now? If not, is there any way to monitor kafka offset that
>> is
>> being processed by Flink? This should be a use case for everyone who uses
>> Flink with Kafka.
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-maili
>> ng-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consume
>> r-Behaviour-tp8257p12663.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>>
>
Re: Flink Kafka Consumer Behaviour
Posted by Stephan Ewen <se...@apache.org>.
Since it is something special to Kafka 0.8, it could be implemented in a
simple addition to the ZooKeeperOffsetHandler used by the
FlinkKafkaConsumer08.
Would you be willing to contribute this? That would certainly help speeding
up the resolution of the issue...
On Fri, Apr 21, 2017 at 2:33 AM, Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> One additional note:
>
> In FlinkKafkaConsumer 0.9+, the current read offset should already exist
> in Flink metrics.
> See https://issues.apache.org/jira/browse/FLINK-4186.
>
> But yes, this is still missing for 0.8, so you need to directly query ZK
> for this.
>
> Cheers,
> Gordon
>
>
> On 21 April 2017 at 8:28:09 AM, Tzu-Li (Gordon) Tai (tzulitai@apache.org)
> wrote:
>
> Hi Sandeep,
>
> It isn’t fixed yet, so I think external tools like the Kafka offset
> checker still won’t work.
> If you’re using 08 and is currently stuck with this issue, you can still
> directly query ZK to get the offsets.
>
> I think for FlinkKafkaConsumer09 the offset is exposed to Flink's metric
> system using Kafka’s own returned metrics, but for 08 this is still missing.
>
> There is this JIRA [1] that aims at exposing consumer lag across all Kafka
> consumer versions to Flink metrics. Perhaps it would make sense to also
> generally expose the offset for all Kafka consumer versions to Flink
> metrics as well.
>
> - Gordon
>
> [1] https://issues.apache.org/jira/browse/FLINK-6109
>
>
> On 19 April 2017 at 5:11:11 AM, sandeep6 (vr1meghashyam@gmail.com) wrote:
>
> Is this fixed now? If not, is there any way to monitor kafka offset that is
> being processed by Flink? This should be a use case for everyone who uses
> Flink with Kafka.
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-
> tp8257p12663.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
>
Re: Flink Kafka Consumer Behaviour
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
One additional note:
In FlinkKafkaConsumer 0.9+, the current read offset should already exist in Flink metrics.
See https://issues.apache.org/jira/browse/FLINK-4186.
But yes, this is still missing for 0.8, so you need to directly query ZK for this.
Cheers,
Gordon
On 21 April 2017 at 8:28:09 AM, Tzu-Li (Gordon) Tai (tzulitai@apache.org) wrote:
Hi Sandeep,
It isn’t fixed yet, so I think external tools like the Kafka offset checker still won’t work.
If you’re using 08 and is currently stuck with this issue, you can still directly query ZK to get the offsets.
I think for FlinkKafkaConsumer09 the offset is exposed to Flink's metric system using Kafka’s own returned metrics, but for 08 this is still missing.
There is this JIRA [1] that aims at exposing consumer lag across all Kafka consumer versions to Flink metrics. Perhaps it would make sense to also generally expose the offset for all Kafka consumer versions to Flink metrics as well.
- Gordon
[1] https://issues.apache.org/jira/browse/FLINK-6109
On 19 April 2017 at 5:11:11 AM, sandeep6 (vr1meghashyam@gmail.com) wrote:
Is this fixed now? If not, is there any way to monitor kafka offset that is
being processed by Flink? This should be a use case for everyone who uses
Flink with Kafka.
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-tp8257p12663.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink Kafka Consumer Behaviour
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Sandeep,
It isn’t fixed yet, so I think external tools like the Kafka offset checker still won’t work.
If you’re using 08 and is currently stuck with this issue, you can still directly query ZK to get the offsets.
I think for FlinkKafkaConsumer09 the offset is exposed to Flink's metric system using Kafka’s own returned metrics, but for 08 this is still missing.
There is this JIRA [1] that aims at exposing consumer lag across all Kafka consumer versions to Flink metrics. Perhaps it would make sense to also generally expose the offset for all Kafka consumer versions to Flink metrics as well.
- Gordon
[1] https://issues.apache.org/jira/browse/FLINK-6109
On 19 April 2017 at 5:11:11 AM, sandeep6 (vr1meghashyam@gmail.com) wrote:
Is this fixed now? If not, is there any way to monitor kafka offset that is
being processed by Flink? This should be a use case for everyone who uses
Flink with Kafka.
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-tp8257p12663.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.