You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Omar Alvarez (JIRA)" <ji...@apache.org> on 2016/03/15 21:18:33 UTC
[jira] [Created] (FLINK-3616) Python API version inconsistencies
Omar Alvarez created FLINK-3616:
-----------------------------------
Summary: Python API version inconsistencies
Key: FLINK-3616
URL: https://issues.apache.org/jira/browse/FLINK-3616
Project: Flink
Issue Type: Bug
Components: Python API
Affects Versions: 0.10.0
Environment: Rocks 6.1 SP1, CentOS release 6.7 (2.6.32-573.7.1.el6.x86_64), java/oraclejdk/1.8.0_45, Python 2.7.9
Reporter: Omar Alvarez
I'm trying to run a python script that uses several packages present in my python 2.7.9 installation. The script is failing with an import error due to discrepacies in the python versions that the flink environment uses. I have two python instalations 2.6 and 2.7.
When I run my Flink python script everything goes fine, it uses python 2.7, until the taskmanager starts executing the job. The problem is that inside the job the python version used is the 2.6 ignoring the systemwide config that points to python 2.7.
Is there anyway to tell flink where to find the desired python version executable?
This is the error:
{code:title=PyExample|borderStyle=solid}
Trajectory loading complete!
03/15/2016 21:17:17 Job execution switched to status RUNNING.
03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to SCHEDULED
03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to DEPLOYING
03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to RUNNING
03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to SCHEDULED
03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to DEPLOYING
03/15/2016 21:17:17 DataSource (ValueSource)(1/1) switched to FINISHED
03/15/2016 21:17:17 MapPartition (PythonMap)(1/1) switched to RUNNING
03/15/2016 21:17:19 MapPartition (PythonMap)(1/1) switched to FAILED
java.lang.Exception: The user defined 'open()' method caused an exception: External process for task MapPartition (PythonMap) terminated prematurely.
Traceback (most recent call last):
File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
import MDAnalysis
ImportError: No module named MDAnalysis
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:484)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: External process for task MapPartition (PythonMap) terminated prematurely.
Traceback (most recent call last):
File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
import MDAnalysis
ImportError: No module named MDAnalysis
at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.startPython(PythonStreamer.java:116)
at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.setupProcess(PythonStreamer.java:58)
at org.apache.flink.languagebinding.api.java.common.streaming.Streamer.open(Streamer.java:67)
at org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.open(PythonMapPartition.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
... 3 more
03/15/2016 21:17:19 Job execution switched to status FAILING.
03/15/2016 21:17:19 DataSink (TextSink)(1/1) switched to CANCELED
03/15/2016 21:17:19 Job execution switched to status FAILED.
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed.
at org.apache.flink.client.program.Client.runBlocking(Client.java:370)
at org.apache.flink.client.program.Client.runBlocking(Client.java:348)
at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:70)
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:804)
at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:117)
at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395)
at org.apache.flink.client.program.Client.runBlocking(Client.java:252)
at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:675)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:326)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:977)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1027)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply$mcV$sp(JobManager.scala:563)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply(JobManager.scala:509)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$5.apply(JobManager.scala:509)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: The user defined 'open()' method caused an exception: External process for task MapPartition (PythonMap) terminated prematurely.
Traceback (most recent call last):
File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
import MDAnalysis
ImportError: No module named MDAnalysis
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:484)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: External process for task MapPartition (PythonMap) terminated prematurely.
Traceback (most recent call last):
File "/tmp/flink-dist-cache-f108fb65-b75b-4b6a-bffd-4b6c5e3a3229/a2016bdf5ba366bbcbd68a6cc1ac537b/flink/plan.py", line 9, in <module>
import MDAnalysis
ImportError: No module named MDAnalysis
at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.startPython(PythonStreamer.java:116)
at org.apache.flink.languagebinding.api.java.python.streaming.PythonStreamer.setupProcess(PythonStreamer.java:58)
at org.apache.flink.languagebinding.api.java.common.streaming.Streamer.open(Streamer.java:67)
at org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.open(PythonMapPartition.java:47)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
... 3 more
The exception above occurred while trying to run your command.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)