You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Tauseef Janvekar <ta...@gmail.com> on 2023/11/27 11:31:55 UTC
Getting a list of tasks for a running job
Dear Team,
How do we get list of tasks that a particular job executes.
If I go toTask Manager then I do not see any tasks. I am also facing the
issue where job manager is not able to access task manager but my jobs are
completing with no issues.
Any help is appreciated.
Thanks,
Tauseef
Re: Getting a list of tasks for a running job
Posted by Tauseef Janvekar <ta...@gmail.com>.
Hi Yuxin,
Added flink user group
Thanks,
Tauseef
On Tue, 28 Nov 2023 at 11:38, Tauseef Janvekar <ta...@gmail.com>
wrote:
> Hi Yuxin,
> We have deployed it on kubernetes using helm chart -
> https://github.com/bitnami/charts/blob/main/bitnami/flink/values.yaml
> We have used ingress and enabled basic authentication -
> https://kubernetes.github.io/ingress-nginx/examples/auth/basic/
>
> No overrides were done on top of basic helm chart installation.
>
>
> The latest logs are below
> 2023-11-28 06:03:03,636 INFO
> org.apache.flink.connector.kafka.source.enumerator.KafkaSourceEnumerator
> [] - Assigning splits to readers {0=[[Partition: aiops-2, StartingOffset:
> 3054, StoppingOffset: -9223372036854775808], [Partition: aiops-0,
> StartingOffset: 2944, StoppingOffset: -9223372036854775808], [Partition:
> aiops-3, StartingOffset: 3006, StoppingOffset: -9223372036854775808], [
> Partition: aiops-1, StartingOffset: 2945, StoppingOffset: -
> 9223372036854775808], [Partition: aiops-4, StartingOffset: 3181,
> StoppingOffset: -9223372036854775808], [Partition: aiops-5, StartingOffset:
> 3154, StoppingOffset: -9223372036854775808]]}
> 2023-11-28 06:03:21,526 INFO org.apache.pekko.remote.transport.
> ProtocolStateActor [] - No response from remote for outbound
> association. Associate timed out after [20000 ms].
> 2023-11-28 06:03:21,539 WARN org.apache.pekko.remote.
> ReliableDeliverySupervisor [] - Association with remote system
> [pekko.tcp://flink-metrics@flink-flink-server-taskmanager:35177] has
> failed, address is now gated for [50] ms. Reason: [Association failed
> with [pekko.tcp://flink-metrics@flink-flink-server-taskmanager:35177]]
> Caused by: [No response from remote for outbound association. Associate
> timed out after [20000 ms].]
> 2023-11-28 06:03:21,559 WARN org.apache.pekko.remote.transport.netty.
> NettyTransport [] - Remote connection to [null] failed with
> org.jboss.netty.channel.ConnectTimeoutException: connection timed out:
> flink-flink-server-taskmanager/172.20.204.52:35177
> 2023-11-28 06:03:43,700 INFO org.apache.pekko.remote.transport.
> ProtocolStateActor [] - No response from remote for outbound
> association. Associate timed out after [20000 ms].
> 2023-11-28 06:03:43,701 WARN org.apache.pekko.remote.
> ReliableDeliverySupervisor [] - Association with remote system
> [pekko.tcp://flink-metrics@flink-flink-server-taskmanager:35177] has
> failed, address is now gated for [50] ms. Reason: [Association failed
> with [pekko.tcp://flink-metrics@flink-flink-server-taskmanager:35177]]
> Caused by: [No response from remote for outbound association. Associate
> timed out after [20000 ms].]
> 2023-11-28 06:03:43,717 WARN org.apache.pekko.remote.transport.netty.
> NettyTransport [] - Remote connection to [null] failed with
> org.jboss.netty.channel.ConnectTimeoutException: connection timed out:
> flink-flink-server-taskmanager/172.20.204.52:35177
> Thanks,
> Tauseef
>
> On Tue, 28 Nov 2023 at 08:07, Yuxin Tan <ta...@gmail.com> wrote:
>
>> Hi, Tauseef,
>>
>> AFAIK, the most common way to get a list of tasks that a particular
>> job executes is through Flink's Web UI or REST API.
>>
>> Using the Flink Web UI:
>> When you run a Flink cluster, a Web UI is launched by default on port
>> 8081 of the JobManager. By accessing this Web UI through a browser,
>> you can see a list of jobs, an overview of each job, and detailed
>> information about a specific job, including the tasks it executes.
>> Using the REST API:
>> For example, to get detailed information about a specific job, you
>> can call the following API: http://<jobmanager-ip>:8081/jobs/<job-id> [1]
>>
>> > facing the issue where job manager is not able to access task
>> manager but my jobs are completing with no issues.
>> This situation indeed seems peculiar. The limited information
>> provided makes it challenging to pinpoint the exact cause. I would
>> recommend examining the network state, reviewing the configurations,
>> and checking both the JobManager and TaskManager
>> logs for any anomalies or error messages
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/rest_api/
>>
>> Best,
>> Yuxin
>>
>>
>> Tauseef Janvekar <ta...@gmail.com> 于2023年11月27日周一 19:32写道：
>>
>>> Dear Team,
>>>
>>> How do we get list of tasks that a particular job executes.
>>> If I go toTask Manager then I do not see any tasks. I am also facing the
>>> issue where job manager is not able to access task manager but my jobs are
>>> completing with no issues.
>>>
>>> Any help is appreciated.
>>>
>>> Thanks,
>>> Tauseef
>>>
>>
Re: Getting a list of tasks for a running job
Posted by Yuxin Tan <ta...@gmail.com>.
Hi, Tauseef,
AFAIK, the most common way to get a list of tasks that a particular
job executes is through Flink's Web UI or REST API.
Using the Flink Web UI:
When you run a Flink cluster, a Web UI is launched by default on port
8081 of the JobManager. By accessing this Web UI through a browser,
you can see a list of jobs, an overview of each job, and detailed
information about a specific job, including the tasks it executes.
Using the REST API:
For example, to get detailed information about a specific job, you
can call the following API: http://<jobmanager-ip>:8081/jobs/<job-id> [1]
> facing the issue where job manager is not able to access task
manager but my jobs are completing with no issues.
This situation indeed seems peculiar. The limited information
provided makes it challenging to pinpoint the exact cause. I would
recommend examining the network state, reviewing the configurations,
and checking both the JobManager and TaskManager
logs for any anomalies or error messages
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/ops/rest_api/
Best,
Yuxin
Tauseef Janvekar <ta...@gmail.com> 于2023年11月27日周一 19:32写道：
> Dear Team,
>
> How do we get list of tasks that a particular job executes.
> If I go toTask Manager then I do not see any tasks. I am also facing the
> issue where job manager is not able to access task manager but my jobs are
> completing with no issues.
>
> Any help is appreciated.
>
> Thanks,
> Tauseef
>