exception while activating calculateThroughp

[Sigalit Eliazov <e....@gmail.com>]

Hi all


We are using the flink k8s operator latest version with flink 1.14 in order
to deploy our pipelines in application mode (one job per cluster, one job
manager + one task manager)

Once in a few minutes I receive the following error in the job manager and
all the tasks are being restarted.

I understand that the throughput is calculator using data size and time
interval

there are cases where this time interval is negative and i am not able to
understand what can cause it or can i affect on it in any configuration



java.lang.IllegalArgumentException: Time should be non negative

at
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.runtime.throughput.ThroughputEMA.calculateThroughput(ThroughputEMA.java:44)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:80)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:792)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$4(StreamTask.java:784)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
~[flink-dist_2.12-1.14.4.jar:1.14.4]

at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
~[flink-dist_2.12-1.14.4.jar:1.14.4]


and idea/comments are welcome

Thanks

Sigalit

Re: exception while activating calculateThroughp

[Lijie Wang <wa...@gmail.com>]

Hi sigalit,
It's a known bug that has been fixed in Flink 1.15.0. See [1] and [2] for
details.

[1] https://issues.apache.org/jira/browse/FLINK-27712
[2] https://issues.apache.org/jira/browse/FLINK-25454

Best,
Lijie

Sigalit Eliazov <e....@gmail.com> 于2022年6月13日周一 20:17写道：

> Hi all
>
>
> We are using the flink k8s operator latest version with flink 1.14 in
> order to deploy our pipelines in application mode (one job per cluster, one
> job manager + one task manager)
>
> Once in a few minutes I receive the following error in the job manager and
> all the tasks are being restarted.
>
> I understand that the throughput is calculator using data size and time
> interval
>
> there are cases where this time interval is negative and i am not able to
> understand what can cause it or can i affect on it in any configuration
>
>
>
> java.lang.IllegalArgumentException: Time should be non negative
>
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.runtime.throughput.ThroughputEMA.calculateThroughput(ThroughputEMA.java:44)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.runtime.throughput.ThroughputCalculator.calculateThroughput(ThroughputCalculator.java:80)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.debloat(StreamTask.java:792)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$4(StreamTask.java:784)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-dist_2.12-1.14.4.jar:1.14.4]
>
>
> and idea/comments are welcome
>
> Thanks
>
> Sigalit
>