You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Chesnay Schepler (JIRA)" <ji...@apache.org> on 2016/03/16 10:43:33 UTC
[jira] [Commented] (FLINK-3616) Python API version inconsistencies
[ https://issues.apache.org/jira/browse/FLINK-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197102#comment-15197102 ]
Chesnay Schepler commented on FLINK-3616:
-----------------------------------------
For python 2 Flink uses the python version that is bound to the command "python". you should be able to modify this behaviour by setting "python.binary.python2" key in the flink-conf.yaml. this key either has to point directly to the binary, or a command that calls the proper version.
it appears that i never got around to documenting this.
> Python API version inconsistencies
> ----------------------------------
>
> Key: FLINK-3616
> URL: https://issues.apache.org/jira/browse/FLINK-3616
> Project: Flink
> Issue Type: Bug
> Components: Python API
> Affects Versions: 0.10.0
> Environment: Rocks 6.1 SP1, CentOS release 6.7 (2.6.32-573.7.1.el6.x86_64), java/oraclejdk/1.8.0_45, Python 2.7.9
> Reporter: Omar Alvarez
> Labels: Python
>
> I'm trying to run a python script that uses several packages present in my python 2.7.9 installation. The script is failing with an import error due to discrepacies in the python versions that the flink environment uses. I have two python instalations 2.6 and 2.7.
> When I run my Flink python script everything goes fine, it uses python 2.7, until the taskmanager starts executing the job. The problem is that inside the job the python version used is the 2.6 ignoring the systemwide config that points to python 2.7.
> Is there anyway to tell flink where to find the desired python version executable?
> This is the error:
> {code:title=PyExample|borderStyle=solid}
> Trajectory loading complete!
> 03/15/2016 21:17:17 Job execution switched to status RUNNING.
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to SCHEDULED
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to DEPLOYING
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to RUNNING
> 03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to SCHEDULED
> 03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to DEPLOYING
> 03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to FINISHED
> 03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to RUNNING
> 03/15/2016 21:17:19 MapPartition (PythonMap)(1/1) switched to FAILED
> java.lang.Exception: The user defined 'open()' method caused an exception: External process for task MapPartition (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:484)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: External process for task MapPartition (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.startPython(PythonStreamer.java:116)
> at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.setupProcess(PythonStreamer.java:58)
> at org.apache.flink.languagebinding.api.java.common.streaming.Streamer.open(Streamer.java:67)
> at org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.open(PythonMapPartition.java:47)
> at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
> ... 3 more
> 03/15/2016 21:17:19 Job execution switched to status FAILING.
> 03/15/2016 21:17:19 DataSink (TextSink)(1/1) switched to CANCELED
> 03/15/2016 21:17:19 Job execution switched to status FAILED.
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
> at org.apache.flink.client.program.Client.runBlocking(Client.java:370)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:348)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
> at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:70)
> at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:804)
> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:117)
> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:85)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:252)
> at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:675)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326)
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:977)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1027)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply$mcV$sp(JobManager.scala:563)
> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply(JobManager.scala:509)
> at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply(JobManager.scala:509)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.Exception: The user defined 'open()' method caused an exception: External process for task MapPartition (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:484)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: External process for task MapPartition (PythonMap) terminated prematurely.
> Traceback (most recent call last):
> File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
> import MDAnalysis
> ImportError: No module named MDAnalysis
> at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.startPython(PythonStreamer.java:116)
> at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.setupProcess(PythonStreamer.java:58)
> at org.apache.flink.languagebinding.api.java.common.streaming.Streamer.open(Streamer.java:67)
> at org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.open(PythonMapPartition.java:47)
> at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
> ... 3 more
> The exception above occurred while trying to run your command.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)