You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Hitesh Tiwari (JIRA)" <ji...@apache.org> on 2018/03/09 11:27:00 UTC

[jira] [Created] (FLINK-8909) pyflink.sh not working with yarn

Hitesh Tiwari created FLINK-8909:
------------------------------------

             Summary: pyflink.sh not working with yarn
                 Key: FLINK-8909
                 URL: https://issues.apache.org/jira/browse/FLINK-8909
             Project: Flink
          Issue Type: Bug
            Reporter: Hitesh Tiwari


Hi,

i want to run the python application from pyflink.sh  with yarn-cluster mode. 

i have added  "-m yarn-cluster -yn 1 " in pyflink.sh. so my pyflink.sh is :

"$FLINK_BIN_DIR"/flink run -m yarn-cluster -yn 1  -v "$FLINK_ROOT_DIR"/lib/flink-python*.jar "$@"

 

Running below command.:

./bin/pyflink.sh /opt/pnda/hitesh/flink-1.4.2/examples/python/WordCount.py

Error:

java.lang.Exception: The user defined 'open()' method caused an exception: An error occurred while copying the file.
 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:485)
 at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: An error occurred while copying the file.
 at org.apache.flink.api.common.cache.DistributedCache.getFile(DistributedCache.java:78)
 at org.apache.flink.python.api.streaming.data.PythonStreamer.startPython(PythonStreamer.java:114)
 at org.apache.flink.python.api.streaming.data.PythonStreamer.open(PythonStreamer.java:100)
 at org.apache.flink.python.api.functions.PythonMapPartition.open(PythonMapPartition.java:53)
 at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481)
 ... 3 more
Caused by: java.io.FileNotFoundException: File /tmp/flink_dc does not exist or the user running Flink ('yarn') has insufficient permissions to access it.
 at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:115)
 at org.apache.flink.runtime.filecache.FileCache.copy(FileCache.java:241)
 at org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:318)
 at org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:302)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 ... 1 more

03/09/2018 11:20:23 Job execution switched to status FAILING.
java.lang.Exception: The user defined 'open()' method caused an exception: An error occurred while copying the file.
 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:485)
 at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: An error occurred while copying the file.
 at org.apache.flink.api.common.cache.DistributedCache.getFile(DistributedCache.java:78)
 at org.apache.flink.python.api.streaming.data.PythonStreamer.startPython(PythonStreamer.java:114)
 at org.apache.flink.python.api.streaming.data.PythonStreamer.open(PythonStreamer.java:100)
 at org.apache.flink.python.api.functions.PythonMapPartition.open(PythonMapPartition.java:53)
 at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
 at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:481)
 ... 3 more
Caused by: java.io.FileNotFoundException: File /tmp/flink_dc does not exist or the user running Flink ('yarn') has insufficient permissions to access it.
 at org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:115)
 at org.apache.flink.runtime.filecache.FileCache.copy(FileCache.java:241)
 at org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:318)
 at org.apache.flink.runtime.filecache.FileCache$CopyProcess.call(FileCache.java:302)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 ... 1 more
03/09/2018 11:20:23 GroupReduce (PythonGroupReducePreStep)(1/1) switched to CANCELED
03/09/2018 11:20:23 CHAIN MapPartition (PythonGroupReduce -> PythonMap) -> Map (PrintSinkPreStep)(1/1) switched to CANCELED
03/09/2018 11:20:23 DataSink (Print to System.out)(1/1) switched to CANCELED
03/09/2018 11:20:23 Job execution switched to status FAILED.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)