You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Matthias J. Sax" <mj...@informatik.hu-berlin.de> on 2015/06/05 13:59:03 UTC
Re: Local Python Test Execution Problem [Bug in Python Layer?]
I just figured out that the missing file is placed at a different location:
There is:
/tmp/flink_data/output
But python looks in
/tmp/users/1000/flink_data/output
("1000" is my Linux user-id)
I guess, the test expects that the file is created from the test base
class and python looks for the file in the wrong place. It uses the
"tempfile" package that return "/tmp/users/1000" instead of "/tmp". It
seems, that "tempfile" returns a different tmp folder for each user,
however, Flink uses "/tmp" only.
I would claim, this is a bug in the python layer.
-Matthias
On 06/05/2015 12:33 PM, Matthias J. Sax wrote:
> Hi,
>
> I have a local setup problem on my Linux that let Python tests fail. For
> some reason, it cannot write to /tmp/ file... (see error message below).
>
> I can resolve the issue with "sudo rm -rf /tmp/*", but this raises other
> problems on my system. Furthermore, the testing problem is back, after
> restarting my machine.
>
> Do you have any idea, what I need to do, to get my configuration right?
>
> -Matthias
>
>> Running org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest
>> log4j:WARN No appenders could be found for logger (org.apache.flink.languagebinding.api.java.python.PythonPlanBinder).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>> Traceback (most recent call last):
>> File "/tmp/flink_plan/plan.py", line 162, in <module>
>> env = get_environment()
>> File "/tmp/flink_plan/flink/plan/Environment.py", line 34, in get_environment
>> return Environment()
>> File "/tmp/flink_plan/flink/plan/Environment.py", line 44, in __init__
>> self._connection = Connection.OneWayBusyBufferingMappedFileConnection()
>> File "/tmp/flink_plan/flink/connection/Connection.py", line 42, in __init__
>> self._output_file = open(tempfile.gettempdir() + "/flink_data/output", "rb+")
>> IOError: [Errno 2] No such file or directory: '/tmp/user/1000/flink_data/output'
>> Traceback (most recent call last):
>> File "/tmp/flink_plan/plan.py", line 162, in <module>
>> env = get_environment()
>> File "/tmp/flink_plan/flink/plan/Environment.py", line 34, in get_environment
>> return Environment()
>> File "/tmp/flink_plan/flink/plan/Environment.py", line 44, in __init__
>> self._connection = Connection.OneWayBusyBufferingMappedFileConnection()
>> File "/tmp/flink_plan/flink/connection/Connection.py", line 42, in __init__
>> self._output_file = open(tempfile.gettempdir() + "/flink_data/output", "rb+")
>> FileNotFoundError: [Errno 2] No such file or directory: '/tmp/user/1000/flink_data/output'
>> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 5.13 sec <<< FAILURE! - in org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest
>> testPython2(org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest) Time elapsed: 2.784 sec <<< ERROR!
>> java.lang.RuntimeException: Plan file caused an error. Check log-files for details.
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.startPython(PythonPlanBinder.java:202)
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:109)
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:84)
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest.testPython2(PythonPlanBinderTest.java:75)
>>
>> testPython3(org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest) Time elapsed: 2.179 sec <<< ERROR!
>> java.lang.RuntimeException: Plan file caused an error. Check log-files for details.
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.startPython(PythonPlanBinder.java:202)
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:109)
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:84)
>> at org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest.testPython3(PythonPlanBinderTest.java:85)
>>
>>
>> Results :
>>
>> Tests in error:
>> PythonPlanBinderTest.testPython2:75 » Runtime Plan file caused an error. Check...
>> PythonPlanBinderTest.testPython3:85 » Runtime Plan file caused an error. Check...
>
>
Re: Local Python Test Execution Problem [Bug in Python Layer?]
Posted by Maximilian Michels <mx...@apache.org>.
Hi Matthias,
Exactly. The Java program gets the temp file path using System.getProperty(
"java.io.tmpdir") while Python uses the "tempfile.gettempdir()" method.
Turns out, they are semantically different in your case. This is one of the
problems with bootstraping a memory mapped file communication.
On my machine, Python and Java actually both return
/var/folders/p0/153pkhvn2vn79b2yszvb64th0000gn/T.
Can you create a JIRA issue?
Kind regards,
Max
On Fri, Jun 5, 2015 at 1:59 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> I just figured out that the missing file is placed at a different location:
>
> There is:
> /tmp/flink_data/output
>
> But python looks in
> /tmp/users/1000/flink_data/output
>
> ("1000" is my Linux user-id)
>
> I guess, the test expects that the file is created from the test base
> class and python looks for the file in the wrong place. It uses the
> "tempfile" package that return "/tmp/users/1000" instead of "/tmp". It
> seems, that "tempfile" returns a different tmp folder for each user,
> however, Flink uses "/tmp" only.
>
> I would claim, this is a bug in the python layer.
>
>
> -Matthias
>
>
> On 06/05/2015 12:33 PM, Matthias J. Sax wrote:
> > Hi,
> >
> > I have a local setup problem on my Linux that let Python tests fail. For
> > some reason, it cannot write to /tmp/ file... (see error message below).
> >
> > I can resolve the issue with "sudo rm -rf /tmp/*", but this raises other
> > problems on my system. Furthermore, the testing problem is back, after
> > restarting my machine.
> >
> > Do you have any idea, what I need to do, to get my configuration right?
> >
> > -Matthias
> >
> >> Running
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest
> >> log4j:WARN No appenders could be found for logger
> (org.apache.flink.languagebinding.api.java.python.PythonPlanBinder).
> >> log4j:WARN Please initialize the log4j system properly.
> >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info.
> >> Traceback (most recent call last):
> >> File "/tmp/flink_plan/plan.py", line 162, in <module>
> >> env = get_environment()
> >> File "/tmp/flink_plan/flink/plan/Environment.py", line 34, in
> get_environment
> >> return Environment()
> >> File "/tmp/flink_plan/flink/plan/Environment.py", line 44, in __init__
> >> self._connection =
> Connection.OneWayBusyBufferingMappedFileConnection()
> >> File "/tmp/flink_plan/flink/connection/Connection.py", line 42, in
> __init__
> >> self._output_file = open(tempfile.gettempdir() +
> "/flink_data/output", "rb+")
> >> IOError: [Errno 2] No such file or directory:
> '/tmp/user/1000/flink_data/output'
> >> Traceback (most recent call last):
> >> File "/tmp/flink_plan/plan.py", line 162, in <module>
> >> env = get_environment()
> >> File "/tmp/flink_plan/flink/plan/Environment.py", line 34, in
> get_environment
> >> return Environment()
> >> File "/tmp/flink_plan/flink/plan/Environment.py", line 44, in __init__
> >> self._connection =
> Connection.OneWayBusyBufferingMappedFileConnection()
> >> File "/tmp/flink_plan/flink/connection/Connection.py", line 42, in
> __init__
> >> self._output_file = open(tempfile.gettempdir() +
> "/flink_data/output", "rb+")
> >> FileNotFoundError: [Errno 2] No such file or directory:
> '/tmp/user/1000/flink_data/output'
> >> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 5.13
> sec <<< FAILURE! - in
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest
> >>
> testPython2(org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest)
> Time elapsed: 2.784 sec <<< ERROR!
> >> java.lang.RuntimeException: Plan file caused an error. Check log-files
> for details.
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.startPython(PythonPlanBinder.java:202)
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:109)
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:84)
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest.testPython2(PythonPlanBinderTest.java:75)
> >>
> >>
> testPython3(org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest)
> Time elapsed: 2.179 sec <<< ERROR!
> >> java.lang.RuntimeException: Plan file caused an error. Check log-files
> for details.
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.startPython(PythonPlanBinder.java:202)
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.runPlan(PythonPlanBinder.java:109)
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:84)
> >> at
> org.apache.flink.languagebinding.api.java.python.PythonPlanBinderTest.testPython3(PythonPlanBinderTest.java:85)
> >>
> >>
> >> Results :
> >>
> >> Tests in error:
> >> PythonPlanBinderTest.testPython2:75 » Runtime Plan file caused an
> error. Check...
> >> PythonPlanBinderTest.testPython3:85 » Runtime Plan file caused an
> error. Check...
> >
> >
>
>