You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Debraj Manna <su...@gmail.com> on 2021/07/14 12:51:17 UTC

Flink 1.13.1 PartitionNotFoundException

Hi

I am observing my flink jobs is failing with the below error

2021-07-14T12:07:00.918Z INFO runtime.executiongraph.Execution
flink-akka.actor.default-dispatcher-29 transitionState:1446
MetricAggregateFunction -> (Sink: LateMetricSink10, Sink: TSDBSink9)
(12/30) (3489393394c13fd1ad85136e11d67deb) switched from RUNNING to FAILED
on container_e12_1626252668574_0001_01_000005 @ platform4 (dataPort=34313).

org.apache.flink.runtime.io.network.partition.PartitionNotFoundException:
Partition
73b5c8ee0c6db91027ce081379f64669#6@7c77e83642e1b954a91dd7b49cf3043b not
found.

        at
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:280)

        at
org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:198)

        at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:515)

        at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.lambda$triggerPartitionStateCheck$1(SingleInputGate.java:845)

        at
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)

        at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)

        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)

        at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)

        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

        at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

        at
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

        at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


Same has been discussed in the old link. But in my case there is hardly any
load in the cluster. Can someone suggest what can cause this error?


http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html


Thanks,

Re: Flink 1.13.1 PartitionNotFoundException

Posted by Debraj Manna <su...@gmail.com>.
I have increased it to 90000 and seems to be running fine. If I see the
failure still when I add some load I will post back in this thread.

On Wed, Jul 14, 2021 at 7:19 PM Debraj Manna <su...@gmail.com>
wrote:

> Yes I forgot to mention in my first email. I have tried increasing taskmanager.network.request-backoff.max to
> 30000 in flink-conf.yaml. But I am getting the same error.
>
> On Wed, Jul 14, 2021 at 7:10 PM Timo Walther <tw...@apache.org> wrote:
>
>> Hi Debraj,
>>
>> I could find quite a few older emails that were suggesting to play
>> around with the `taskmanager.network.request-backoff.max` option. This
>> was also recomended in the link that you shared. Have you tried it?
>>
>> Here is some background:
>>
>>
>> http://deprecated-apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Network-PartitionNotFoundException-when-run-on-multi-nodes-td21683.html
>>
>> https://issues.apache.org/jira/browse/FLINK-9413
>>
>> Let us know if it helped.
>>
>> Regards,
>> Timo
>>
>>
>>
>> On 14.07.21 14:51, Debraj Manna wrote:
>> > Hi
>> >
>> > I am observing my flink jobs is failing with the below error
>> >
>> > 2021-07-14T12:07:00.918Z INFO runtime.executiongraph.Execution
>> > flink-akka.actor.default-dispatcher-29 transitionState:1446
>> > MetricAggregateFunction -> (Sink: LateMetricSink10, Sink: TSDBSink9)
>> > (12/30) (3489393394c13fd1ad85136e11d67deb) switched from RUNNING to
>> > FAILED on container_e12_1626252668574_0001_01_000005 @ platform4
>> > (dataPort=34313).
>> >
>> > org.apache.flink.runtime.io.network.partition.PartitionNotFoundException:
>> Partition
>> > 73b5c8ee0c6db91027ce081379f64669#6@7c77e83642e1b954a91dd7b49cf3043b
>> not
>> > found.
>> >
>> > at
>> > org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:280)
>> >
>> > at
>> > org.apache.flink.runtime.io
>> .network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:198)
>> >
>> > at
>> > org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:515)
>> >
>> > at
>> > org.apache.flink.runtime.io
>> .network.partition.consumer.SingleInputGate.lambda$triggerPartitionStateCheck$1(SingleInputGate.java:845)
>> >
>> > at
>> >
>> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
>> >
>> > at
>> >
>> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
>> >
>> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
>> >
>> > at
>> >
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
>> >
>> > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>> >
>> > at
>> >
>> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>> >
>> > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> >
>> > at
>> >
>> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >
>> >
>> > Same has been discussed in the old link. But in my case there is hardly
>> > any load in the cluster. Can someone suggest what can cause this error?
>> >
>> >
>> >
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html
>> > <
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html
>> >
>> >
>> >
>> > Thanks,
>> >
>> >
>>
>>

Re: Flink 1.13.1 PartitionNotFoundException

Posted by Debraj Manna <su...@gmail.com>.
Yes I forgot to mention in my first email. I have tried increasing
taskmanager.network.request-backoff.max to
30000 in flink-conf.yaml. But I am getting the same error.

On Wed, Jul 14, 2021 at 7:10 PM Timo Walther <tw...@apache.org> wrote:

> Hi Debraj,
>
> I could find quite a few older emails that were suggesting to play
> around with the `taskmanager.network.request-backoff.max` option. This
> was also recomended in the link that you shared. Have you tried it?
>
> Here is some background:
>
>
> http://deprecated-apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Network-PartitionNotFoundException-when-run-on-multi-nodes-td21683.html
>
> https://issues.apache.org/jira/browse/FLINK-9413
>
> Let us know if it helped.
>
> Regards,
> Timo
>
>
>
> On 14.07.21 14:51, Debraj Manna wrote:
> > Hi
> >
> > I am observing my flink jobs is failing with the below error
> >
> > 2021-07-14T12:07:00.918Z INFO runtime.executiongraph.Execution
> > flink-akka.actor.default-dispatcher-29 transitionState:1446
> > MetricAggregateFunction -> (Sink: LateMetricSink10, Sink: TSDBSink9)
> > (12/30) (3489393394c13fd1ad85136e11d67deb) switched from RUNNING to
> > FAILED on container_e12_1626252668574_0001_01_000005 @ platform4
> > (dataPort=34313).
> >
> > org.apache.flink.runtime.io.network.partition.PartitionNotFoundException:
> Partition
> > 73b5c8ee0c6db91027ce081379f64669#6@7c77e83642e1b954a91dd7b49cf3043b not
> > found.
> >
> > at
> > org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:280)
> >
> > at
> > org.apache.flink.runtime.io
> .network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:198)
> >
> > at
> > org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:515)
> >
> > at
> > org.apache.flink.runtime.io
> .network.partition.consumer.SingleInputGate.lambda$triggerPartitionStateCheck$1(SingleInputGate.java:845)
> >
> > at
> >
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
> >
> > at
> >
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> >
> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> >
> > at
> >
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
> >
> > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> >
> > at
> >
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> >
> > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >
> > at
> >
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> >
> > Same has been discussed in the old link. But in my case there is hardly
> > any load in the cluster. Can someone suggest what can cause this error?
> >
> >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html
> > <
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html
> >
> >
> >
> > Thanks,
> >
> >
>
>

Re: Flink 1.13.1 PartitionNotFoundException

Posted by Timo Walther <tw...@apache.org>.
Hi Debraj,

I could find quite a few older emails that were suggesting to play 
around with the `taskmanager.network.request-backoff.max` option. This 
was also recomended in the link that you shared. Have you tried it?

Here is some background:

http://deprecated-apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Network-PartitionNotFoundException-when-run-on-multi-nodes-td21683.html

https://issues.apache.org/jira/browse/FLINK-9413

Let us know if it helped.

Regards,
Timo



On 14.07.21 14:51, Debraj Manna wrote:
> Hi
> 
> I am observing my flink jobs is failing with the below error
> 
> 2021-07-14T12:07:00.918Z INFO runtime.executiongraph.Execution 
> flink-akka.actor.default-dispatcher-29 transitionState:1446 
> MetricAggregateFunction -> (Sink: LateMetricSink10, Sink: TSDBSink9) 
> (12/30) (3489393394c13fd1ad85136e11d67deb) switched from RUNNING to 
> FAILED on container_e12_1626252668574_0001_01_000005 @ platform4 
> (dataPort=34313).
> 
> org.apache.flink.runtime.io.network.partition.PartitionNotFoundException: Partition 
> 73b5c8ee0c6db91027ce081379f64669#6@7c77e83642e1b954a91dd7b49cf3043b not 
> found.
> 
> at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:280)
> 
> at 
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:198)
> 
> at 
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:515)
> 
> at 
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.lambda$triggerPartitionStateCheck$1(SingleInputGate.java:845)
> 
> at 
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
> 
> at 
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> 
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> 
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
> 
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> 
> at 
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> 
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 
> at 
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 
> 
> Same has been discussed in the old link. But in my case there is hardly 
> any load in the cluster. Can someone suggest what can cause this error?
> 
> 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html 
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html>
> 
> 
> Thanks,
> 
>