You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Junrui Lee <jr...@gmail.com> on 2023/11/06 05:23:30 UTC
Re: Flink Job Failed With Kafka Exception
Hi Madan,
Do you mean you want to restart only the failed tasks, rather than
restarting the entire pipeline region? As far as I know, currently Flink
does not support task-level restart, but requires restarting the pipeline
region.
Best,
Junrui
Madan D via user <us...@flink.apache.org> 于2023年10月11日周三 12:37写道：
> Hello Team,
> We are running the Flink pipeline by consuming data from multiple topics,
> but we recently encountered that if there's one topic having issues with
> participation, etc., the whole Flink pipeline is failing, which is
> affecting topics. Is there a way we can make Flink Piplein keep running
> even after one of the topics has an issue? We tried to handle exceptions to
> make sure the job wouldn't fail, but it didn't help out.
>
> Caused by: java.lang.RuntimeException: Failed to get metadata for topics
>
>
> Can you please provide any insights?
>
>
> Regards,
> Madan
>
Re: Flink Job Failed With Kafka Exception
Posted by Feng Jin <ji...@gmail.com>.
Hi Madan,
Perhaps you can filter out inactive topics in the client first and then
pass the filtered list of topics to KafkaConsumer.
Best,
Feng
On Tue, Nov 7, 2023 at 10:42 AM Madan D via user <us...@flink.apache.org>
wrote:
> Hello Hang/Lee,
> Thanks!
> In my usecase we listen from multiple topics but in few cases one of the
> topic may become inactive if producer decides to shutdown one of the topic
> but other topics still will be receiving data but what we observe is that
> if there’s one of the topic is getting in-active entire flink application
> is getting failed due to time out while getting metadata but we would like
> flink job to continue to consume data from other source topics even if one
> of the topic has any issue since failing entire flink application doesn’t
> make sense if one if the topic has issue.
>
>
>
> Regards,
> Madan
>
> On Nov 5, 2023, at 11:29 PM, Hang Ruan <ru...@gmail.com> wrote:
>
>
> Hi, Madan.
>
> This error seems like that there are some problems when the consumer tries
> to read the topic metadata. If you use the same source for these topics,
> the kafka connector cannot skip one of them. As you say, you need to modify
> the connector's default behavior.
> Maybe you should read the code in KafkaSourceEnumerator to skip this error.
>
> Best,
> Hang
>
> Junrui Lee <jr...@gmail.com> 于2023年11月6日周一 14:30写道：
>
>> Hi Madan,
>>
>> Do you mean you want to restart only the failed tasks, rather than
>> restarting the entire pipeline region? As far as I know, currently Flink
>> does not support task-level restart, but requires restarting the pipeline
>> region.
>>
>> Best,
>> Junrui
>>
>> Madan D via user <us...@flink.apache.org> 于2023年10月11日周三 12:37写道：
>>
>>> Hello Team,
>>> We are running the Flink pipeline by consuming data from multiple
>>> topics, but we recently encountered that if there's one topic having issues
>>> with participation, etc., the whole Flink pipeline is failing, which is
>>> affecting topics. Is there a way we can make Flink Piplein keep running
>>> even after one of the topics has an issue? We tried to handle exceptions to
>>> make sure the job wouldn't fail, but it didn't help out.
>>>
>>> Caused by: java.lang.RuntimeException: Failed to get metadata for topics
>>>
>>>
>>> Can you please provide any insights?
>>>
>>>
>>> Regards,
>>> Madan
>>>
>>
Re: Flink Job Failed With Kafka Exception
Posted by Madan D via user <us...@flink.apache.org>.
Hello Hang/Lee,
Thanks!
In my usecase we listen from multiple topics but in few cases one of the topic
may become inactive if producer decides to shutdown one of the topic but other
topics still will be receiving data but what we observe is that if there’s one
of the topic is getting in-active entire flink application is getting failed
due to time out while getting metadata but we would like flink job to continue
to consume data from other source topics even if one of the topic has any
issue since failing entire flink application doesn’t make sense if one if the
topic has issue.
Regards,
Madan
> On Nov 5, 2023, at 11:29 PM, Hang Ruan <ru...@gmail.com> wrote:
>
>
>
>
> Hi, Madan.
>
>
>
>
> This error seems like that there are some problems when the consumer tries
> to read the topic metadata. If you use the same source for these topics, the
> kafka connector cannot skip one of them. As you say, you need to modify the
> connector's default behavior.
>
> Maybe you should read the code in KafkaSourceEnumerator to skip this error.
>
>
>
>
> Best,
>
> Hang
>
>
>
>
> Junrui Lee <[jrlee.ljr@gmail.com](mailto:jrlee.ljr@gmail.com)> 于2023年11月6日周一
> 14:30写道：
>
>
>> Hi Madan,
>
> Do you mean you want to restart only the failed tasks, rather than
> restarting the entire pipeline region? As far as I know, currently Flink
> does not support task-level restart, but requires restarting the pipeline
> region.
>
> Best,
> Junrui
>
>>
>>
>
>>
>> Madan D via user <[user@flink.apache.org](mailto:user@flink.apache.org)>
于2023年10月11日周三 12:37写道：
>
>>
>>> Hello Team,
>>>
>>> We are running the Flink pipeline by consuming data from multiple topics,
but we recently encountered that if there's one topic having issues with
participation, etc., the whole Flink pipeline is failing, which is affecting
topics. Is there a way we can make Flink Piplein keep running even after one
of the topics has an issue? We tried to handle exceptions to make sure the job
wouldn't fail, but it didn't help out.
>>>
>>>
>
>>>
>>>
>>> Caused by: java.lang.RuntimeException: Failed to get metadata for
topics
>>>
>>>
>>>
>>> Can you please provide any insights?
>>>
>>>
>
>>>
>>>
>
>>>
>>> Regards,
>>>
>>> Madan
Re: Flink Job Failed With Kafka Exception
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Madan.
This error seems like that there are some problems when the consumer tries
to read the topic metadata. If you use the same source for these topics,
the kafka connector cannot skip one of them. As you say, you need to modify
the connector's default behavior.
Maybe you should read the code in KafkaSourceEnumerator to skip this error.
Best,
Hang
Junrui Lee <jr...@gmail.com> 于2023年11月6日周一 14:30写道：
> Hi Madan,
>
> Do you mean you want to restart only the failed tasks, rather than
> restarting the entire pipeline region? As far as I know, currently Flink
> does not support task-level restart, but requires restarting the pipeline
> region.
>
> Best,
> Junrui
>
> Madan D via user <us...@flink.apache.org> 于2023年10月11日周三 12:37写道：
>
>> Hello Team,
>> We are running the Flink pipeline by consuming data from multiple topics,
>> but we recently encountered that if there's one topic having issues with
>> participation, etc., the whole Flink pipeline is failing, which is
>> affecting topics. Is there a way we can make Flink Piplein keep running
>> even after one of the topics has an issue? We tried to handle exceptions to
>> make sure the job wouldn't fail, but it didn't help out.
>>
>> Caused by: java.lang.RuntimeException: Failed to get metadata for topics
>>
>>
>> Can you please provide any insights?
>>
>>
>> Regards,
>> Madan
>>
>