You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Nathan Moderwell <na...@robinhood.com> on 2023/05/04 19:54:06 UTC
Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)
Hi,
I'm running the sql-runner example in the flink-kuberenetes-operator repo
and hitting an error. I've tried building the image from different releases
of the repo and that does not help. The pods run (status stays at Running)
and it is able to build the execution graph successfully, however I see the
error below (the important part seems to be `The file STDOUT does not exist
on the TaskExecutor`). I've also tried building the python example that has
a similar sql query and I get the same error. I'm running this example on
minikube, building everything as-is from the OSS flink-kubernetes-operator
repo. Any idea why I'm getting this error?
Full Error Logs:
2023-05-04 19:36:29,983 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
orders[1] -> Sink: print_table[2] (1/1)
(fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from SCHEDULED to DEPLOYING.
2023-05-04 19:36:30,092 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Deploying Source: orders[1] -> Sink: print_table[2] (1/1) (attempt #0) with
attempt id
fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and
vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to
python-example-taskmanager-1-1 @ 10.244.0.40 (dataPort=39441) with
allocation id fe50815410ace91f428c4aee34bce7ab
2023-05-04 19:36:30,388 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
orders[1] -> Sink: print_table[2] (1/1)
(fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from DEPLOYING to INITIALIZING.
2023-05-04 19:36:30,496 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
orders[1] -> Sink: print_table[2] (1/1)
(fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
switched from INITIALIZING to RUNNING.
2023-05-04 19:39:23,567 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
[] - Failed to transfer file from TaskExecutor
python-example-taskmanager-1-1.
java.util.concurrent.CompletionException:
org.apache.flink.util.FlinkException: The file STDOUT does not exist on the
TaskExecutor.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:261)
~[?:?]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1275)
~[flink-dist-1.16.1.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.OnComplete.internal(Future.scala:299)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.OnComplete.internal(Future.scala:297)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:25)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
[?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
[?:?]
at
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[?:?]
Caused by: org.apache.flink.util.FlinkException: The file STDOUT does not
exist on the TaskExecutor.
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
~[flink-dist-1.16.1.jar:1.16.1]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-05-04 19:39:23,576 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
[] - Unhandled exception.
org.apache.flink.util.FlinkException: The file STDOUT does not exist on the
TaskExecutor.
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
~[flink-dist-1.16.1.jar:1.16.1]
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Re: Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)
Posted by Nathan Moderwell <na...@robinhood.com>.
Thanks Hang and Gyula for the quick response! That makes sense, I can
resolve manually for now.
On Thu, May 4, 2023 at 8:37 PM Hang Ruan <ru...@gmail.com> wrote:
> Hi, Nathan,
>
> I think the previous discussion[1] is helpful for you.
>
> Best,
> Hang
>
> [1] https://lists.apache.org/thread/cgwsr6f1l3202ktwcvtyxtdsgj2vyms1
>
> Nathan Moderwell <na...@robinhood.com> 于2023年5月5日周五 03:54写道：
>
>> Hi,
>>
>> I'm running the sql-runner example in the flink-kuberenetes-operator repo
>> and hitting an error. I've tried building the image from different releases
>> of the repo and that does not help. The pods run (status stays at Running)
>> and it is able to build the execution graph successfully, however I see the
>> error below (the important part seems to be `The file STDOUT does not exist
>> on the TaskExecutor`). I've also tried building the python example that has
>> a similar sql query and I get the same error. I'm running this example on
>> minikube, building everything as-is from the OSS flink-kubernetes-operator
>> repo. Any idea why I'm getting this error?
>>
>> Full Error Logs:
>>
>> 2023-05-04 19:36:29,983 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
>> orders[1] -> Sink: print_table[2] (1/1)
>> (fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
>> switched from SCHEDULED to DEPLOYING.
>> 2023-05-04 19:36:30,092 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
>> Deploying Source: orders[1] -> Sink: print_table[2] (1/1) (attempt #0) with
>> attempt id
>> fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and
>> vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to
>> python-example-taskmanager-1-1 @ 10.244.0.40 (dataPort=39441) with
>> allocation id fe50815410ace91f428c4aee34bce7ab
>> 2023-05-04 19:36:30,388 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
>> orders[1] -> Sink: print_table[2] (1/1)
>> (fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
>> switched from DEPLOYING to INITIALIZING.
>> 2023-05-04 19:36:30,496 INFO
>> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
>> orders[1] -> Sink: print_table[2] (1/1)
>> (fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
>> switched from INITIALIZING to RUNNING.
>> 2023-05-04 19:39:23,567 ERROR
>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
>> [] - Failed to transfer file from TaskExecutor
>> python-example-taskmanager-1-1.
>> java.util.concurrent.CompletionException:
>> org.apache.flink.util.FlinkException: The file STDOUT does not exist on the
>> TaskExecutor.
>> at
>> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
>> ~[?:?]
>> at
>> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:261)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
>> ~[?:?]
>> at
>> org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1275)
>> ~[flink-dist-1.16.1.jar:1.16.1]
>> at
>> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>> ~[?:?]
>> at
>> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
>> ~[?:?]
>> at
>> org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at akka.dispatch.OnComplete.internal(Future.scala:299)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at akka.dispatch.OnComplete.internal(Future.scala:297)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:25)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
>> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
>> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
>> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
>> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
>> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
>> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
>> at
>> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
>> at
>> java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
>> [?:?]
>> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
>> [?:?]
>> at
>> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
>> at
>> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
>> [?:?]
>> Caused by: org.apache.flink.util.FlinkException: The file STDOUT does not
>> exist on the TaskExecutor.
>> at
>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
>> ~[flink-dist-1.16.1.jar:1.16.1]
>> at
>> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> ~[?:?]
>> at java.lang.Thread.run(Thread.java:829) ~[?:?]
>> 2023-05-04 19:39:23,576 ERROR
>> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
>> [] - Unhandled exception.
>> org.apache.flink.util.FlinkException: The file STDOUT does not exist on
>> the TaskExecutor.
>> at
>> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
>> ~[flink-dist-1.16.1.jar:1.16.1]
>> at
>> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> ~[?:?]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> ~[?:?]
>> at java.lang.Thread.run(Thread.java:829) ~[?:?]
>>
>>
>>
--
<http://www.robinhood.com/>
Nathan Moderwell
Senior Machine Learning Engineer
Menlo Park, CA
Don't copy, share, or use this email without permission. If you received it
by accident, please let us know and then delete it right away.
Re: Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Nathan,
I think the previous discussion[1] is helpful for you.
Best,
Hang
[1] https://lists.apache.org/thread/cgwsr6f1l3202ktwcvtyxtdsgj2vyms1
Nathan Moderwell <na...@robinhood.com> 于2023年5月5日周五 03:54写道：
> Hi,
>
> I'm running the sql-runner example in the flink-kuberenetes-operator repo
> and hitting an error. I've tried building the image from different releases
> of the repo and that does not help. The pods run (status stays at Running)
> and it is able to build the execution graph successfully, however I see the
> error below (the important part seems to be `The file STDOUT does not exist
> on the TaskExecutor`). I've also tried building the python example that has
> a similar sql query and I get the same error. I'm running this example on
> minikube, building everything as-is from the OSS flink-kubernetes-operator
> repo. Any idea why I'm getting this error?
>
> Full Error Logs:
>
> 2023-05-04 19:36:29,983 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
> orders[1] -> Sink: print_table[2] (1/1)
> (fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
> switched from SCHEDULED to DEPLOYING.
> 2023-05-04 19:36:30,092 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
> Deploying Source: orders[1] -> Sink: print_table[2] (1/1) (attempt #0) with
> attempt id
> fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0 and
> vertex id cbc357ccb763df2852fee8c4fc7d55f2_0 to
> python-example-taskmanager-1-1 @ 10.244.0.40 (dataPort=39441) with
> allocation id fe50815410ace91f428c4aee34bce7ab
> 2023-05-04 19:36:30,388 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
> orders[1] -> Sink: print_table[2] (1/1)
> (fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
> switched from DEPLOYING to INITIALIZING.
> 2023-05-04 19:36:30,496 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
> orders[1] -> Sink: print_table[2] (1/1)
> (fa0a897ca85acd1d2b7719bb2c929d2f_cbc357ccb763df2852fee8c4fc7d55f2_0_0)
> switched from INITIALIZING to RUNNING.
> 2023-05-04 19:39:23,567 ERROR
> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
> [] - Failed to transfer file from TaskExecutor
> python-example-taskmanager-1-1.
> java.util.concurrent.CompletionException:
> org.apache.flink.util.FlinkException: The file STDOUT does not exist on the
> TaskExecutor.
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> ~[?:?]
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:261)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> ~[?:?]
> at
> org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1275)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> ~[?:?]
> at
> org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at akka.dispatch.OnComplete.internal(Future.scala:299)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at akka.dispatch.OnComplete.internal(Future.scala:297)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:25)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
> ~[flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
> [flink-rpc-akka_02bdeb39-d7bb-42a6-9466-c8eb2b8bfca2.jar:1.16.1]
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
> [?:?]
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
> [?:?]
> at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
> [?:?]
> at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
> [?:?]
> Caused by: org.apache.flink.util.FlinkException: The file STDOUT does not
> exist on the TaskExecutor.
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:829) ~[?:?]
> 2023-05-04 19:39:23,576 ERROR
> org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler
> [] - Unhandled exception.
> org.apache.flink.util.FlinkException: The file STDOUT does not exist on
> the TaskExecutor.
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.lambda$requestFileUploadByFilePath$24(TaskExecutor.java:2154)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:829) ~[?:?]
>
>
>