You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Peter Vary (Jira)" <ji...@apache.org> on 2022/12/05 12:09:00 UTC
[jira] [Commented] (FLINK-30150) Evaluate operator error log whitelist entry: REST service in session cluster is bad now
[ https://issues.apache.org/jira/browse/FLINK-30150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643305#comment-17643305 ]
Peter Vary commented on FLINK-30150:
------------------------------------
This is the exception in the logs:
{code:java}
2022-12-05T11:40:59.2665289Z [m[33m2022-12-05 11:40:26,746[m [36mo.a.f.k.o.o.d.SessionObserver [m [1;31m[ERROR][default/session-cluster-1] REST service in session cluster is bad now
2022-12-05T11:40:59.2665851Z java.util.concurrent.TimeoutException
2022-12-05T11:40:59.2666258Z at java.base/java.util.concurrent.CompletableFuture.timedGet(Unknown Source)
2022-12-05T11:40:59.2666841Z at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
2022-12-05T11:40:59.2667549Z at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.listJobs(AbstractFlinkService.java:231)
2022-12-05T11:40:59.2668462Z at org.apache.flink.kubernetes.operator.observer.deployment.SessionObserver.observeFlinkCluster(SessionObserver.java:48)
2022-12-05T11:40:59.2669809Z at org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:89)
2022-12-05T11:40:59.2671385Z at org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:55)
2022-12-05T11:40:59.2672514Z at org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:56)
2022-12-05T11:40:59.2673507Z at org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:32)
2022-12-05T11:40:59.2674466Z at org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:113)
2022-12-05T11:40:59.2675692Z at org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:54)
2022-12-05T11:40:59.2676509Z at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:136)
2022-12-05T11:40:59.2677043Z at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:94)
2022-12-05T11:40:59.2677741Z at org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
2022-12-05T11:40:59.2678451Z at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:93)
2022-12-05T11:40:59.2679180Z at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:130)
2022-12-05T11:40:59.2680055Z at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:110)
2022-12-05T11:40:59.2681621Z at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:81)
2022-12-05T11:40:59.2682478Z at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:54)
2022-12-05T11:40:59.2683241Z at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
2022-12-05T11:40:59.2683817Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2022-12-05T11:40:59.2684294Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2022-12-05T11:40:59.2684676Z at java.base/java.lang.Thread.run(Unknown Source) {code}
The log line show 2022-12-05 11:40:26,746 as the timestamp.
This is happening when we manually kill the job to test the recovery:
{code:java}
2022-12-05T11:40:12.8330378Z Successfully verified that sessionjob/flink-example-statemachine.status.jobStatus.state is in RUNNING state.
2022-12-05T11:40:12.9711940Z Kill the session-cluster-1-7bc5b4d7cb-t5hgq
2022-12-05T11:40:13.3083721Z Waiting for log "Restoring job ffffffff9b85cb750000000000000001 from Checkpoint"...
2022-12-05T11:40:35.8208688Z Log "Restoring job ffffffff9b85cb750000000000000001 from Checkpoint" shows up. {code}
I would say that this is expected.
> Evaluate operator error log whitelist entry: REST service in session cluster is bad now
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-30150
> URL: https://issues.apache.org/jira/browse/FLINK-30150
> Project: Flink
> Issue Type: Sub-task
> Components: Kubernetes Operator
> Reporter: Gabor Somogyi
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)