You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Chesnay Schepler <ch...@apache.org> on 2016/01/27 18:05:47 UTC
Python start scripts broken
Hello everyone,
I just noticed an odd behavior that occurs when using the python api
start scripts: i suddenly get another argument i didn't get before; the
flink-python jar path. which inadvertently breaks everything.
An example as to how the execution of an example should go:
bin\pyflink2.sh resources\python\flink\example\Wordcount.py
(1 argument)
the pyflink2.sh script then calls:
bin\flink -v libraries\<python-jar> 2(this is the python version)
resources\python\flink\example\Wordcount.py
(3 arguments)
which is supposed to pass the arguments "2" and
"resources\python\flink\example\Wordcount.py" to the PythonPlanBinder's
main method.
Now for some reason the main method gets 3 arguments, the path to the
python jar, and the 2 arguments it is supposed to get.
I haven't changed the scripts in a long time, so i assume for now that
an external change is the cause of this. Did anyone work on something
that could cause this?
Re: Python start scripts broken
Posted by Chesnay Schepler <ch...@apache.org>.
Running mvn clean package resolved the issue. still the weirdest thing
that happened this week. :/
On 27.01.2016 20:19, Chesnay Schepler wrote:
> i went back 30 commits and the issue is gone, as such its not a system
> problem. gonna narrow it down now.
>
> On 27.01.2016 19:29, Chesnay Schepler wrote:
>> Yes i am on windows. no, ain't no path containing spaces.
>>
>> if it were due to spaces i would assume that either a) the python jar
>> wouldn't be called properly or b) any of the arguments would be
>> malformed, but they aren't.
>>
>> the following are the actual arguments that are passed to the
>> PythonPlanBinder main method as printed by Arrays.toString():
>>
>> [C:\Users\Zento\Documents\GitHub\flink\flink-dist\target\flink-1.0-SNAPSHOT-bin\flink-1.0-SNAPSHOT\lib\flink-python_2.10-1.0-SNAPSHOT.jar,
>> 2, resources\python\flink\example\WordCount.py]
>>
>> Supposedly, these are the exact arguments passed to bin\flink .
>>
>> i could just ignore the additional argument and have it work again,
>> but i would still like to know why that argument even arrives here.
>>
>> On 27.01.2016 18:58, Robert Metzger wrote:
>>> Given the direction of the path separators, I guess you are using
>>> Microsoft
>>> Windows. I think Fabian is very experienced fixing windows path issues
>>> before every release ... and usually the issue are paths containing
>>> spaces.
>>> Maybe that's the case here as well?
>>>
>>> On Wed, Jan 27, 2016 at 6:51 PM, Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>>> I was trending more towards changes to the CLI.
>>>>
>>>>
>>>> On 27.01.2016 18:48, Robert Metzger wrote:
>>>>
>>>>> Well, according to the git history, it was only me who touched the
>>>>> file
>>>>> many months ago:
>>>>>
>>>>> https://github.com/apache/flink/commits/master/flink-dist/src/main/flink-bin/bin/pyflink2.sh
>>>>>
>>>>>
>>>>> On Wed, Jan 27, 2016 at 6:05 PM, Chesnay Schepler
>>>>> <ch...@apache.org>
>>>>> wrote:
>>>>>
>>>>> Hello everyone,
>>>>>> I just noticed an odd behavior that occurs when using the python api
>>>>>> start
>>>>>> scripts: i suddenly get another argument i didn't get before; the
>>>>>> flink-python jar path. which inadvertently breaks everything.
>>>>>>
>>>>>> An example as to how the execution of an example should go:
>>>>>>
>>>>>> bin\pyflink2.sh resources\python\flink\example\Wordcount.py
>>>>>> (1 argument)
>>>>>>
>>>>>> the pyflink2.sh script then calls:
>>>>>>
>>>>>> bin\flink -v libraries\<python-jar> 2(this is the python version)
>>>>>> resources\python\flink\example\Wordcount.py
>>>>>> (3 arguments)
>>>>>>
>>>>>> which is supposed to pass the arguments "2" and
>>>>>> "resources\python\flink\example\Wordcount.py" to the
>>>>>> PythonPlanBinder's
>>>>>> main method.
>>>>>>
>>>>>> Now for some reason the main method gets 3 arguments, the path to
>>>>>> the
>>>>>> python jar, and the 2 arguments it is supposed to get.
>>>>>>
>>>>>> I haven't changed the scripts in a long time, so i assume for now
>>>>>> that an
>>>>>> external change is the cause of this. Did anyone work on
>>>>>> something that
>>>>>> could cause this?
>>>>>>
>>>>>>
>>>>>>
>>
>>
>
>
Re: Python start scripts broken
Posted by Chesnay Schepler <ch...@apache.org>.
i went back 30 commits and the issue is gone, as such its not a system
problem. gonna narrow it down now.
On 27.01.2016 19:29, Chesnay Schepler wrote:
> Yes i am on windows. no, ain't no path containing spaces.
>
> if it were due to spaces i would assume that either a) the python jar
> wouldn't be called properly or b) any of the arguments would be
> malformed, but they aren't.
>
> the following are the actual arguments that are passed to the
> PythonPlanBinder main method as printed by Arrays.toString():
>
> [C:\Users\Zento\Documents\GitHub\flink\flink-dist\target\flink-1.0-SNAPSHOT-bin\flink-1.0-SNAPSHOT\lib\flink-python_2.10-1.0-SNAPSHOT.jar,
> 2, resources\python\flink\example\WordCount.py]
>
> Supposedly, these are the exact arguments passed to bin\flink .
>
> i could just ignore the additional argument and have it work again,
> but i would still like to know why that argument even arrives here.
>
> On 27.01.2016 18:58, Robert Metzger wrote:
>> Given the direction of the path separators, I guess you are using
>> Microsoft
>> Windows. I think Fabian is very experienced fixing windows path issues
>> before every release ... and usually the issue are paths containing
>> spaces.
>> Maybe that's the case here as well?
>>
>> On Wed, Jan 27, 2016 at 6:51 PM, Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> I was trending more towards changes to the CLI.
>>>
>>>
>>> On 27.01.2016 18:48, Robert Metzger wrote:
>>>
>>>> Well, according to the git history, it was only me who touched the
>>>> file
>>>> many months ago:
>>>>
>>>> https://github.com/apache/flink/commits/master/flink-dist/src/main/flink-bin/bin/pyflink2.sh
>>>>
>>>>
>>>> On Wed, Jan 27, 2016 at 6:05 PM, Chesnay Schepler <ch...@apache.org>
>>>> wrote:
>>>>
>>>> Hello everyone,
>>>>> I just noticed an odd behavior that occurs when using the python api
>>>>> start
>>>>> scripts: i suddenly get another argument i didn't get before; the
>>>>> flink-python jar path. which inadvertently breaks everything.
>>>>>
>>>>> An example as to how the execution of an example should go:
>>>>>
>>>>> bin\pyflink2.sh resources\python\flink\example\Wordcount.py
>>>>> (1 argument)
>>>>>
>>>>> the pyflink2.sh script then calls:
>>>>>
>>>>> bin\flink -v libraries\<python-jar> 2(this is the python version)
>>>>> resources\python\flink\example\Wordcount.py
>>>>> (3 arguments)
>>>>>
>>>>> which is supposed to pass the arguments "2" and
>>>>> "resources\python\flink\example\Wordcount.py" to the
>>>>> PythonPlanBinder's
>>>>> main method.
>>>>>
>>>>> Now for some reason the main method gets 3 arguments, the path to the
>>>>> python jar, and the 2 arguments it is supposed to get.
>>>>>
>>>>> I haven't changed the scripts in a long time, so i assume for now
>>>>> that an
>>>>> external change is the cause of this. Did anyone work on something
>>>>> that
>>>>> could cause this?
>>>>>
>>>>>
>>>>>
>
>
Re: Python start scripts broken
Posted by Chesnay Schepler <ch...@apache.org>.
Yes i am on windows. no, ain't no path containing spaces.
if it were due to spaces i would assume that either a) the python jar
wouldn't be called properly or b) any of the arguments would be
malformed, but they aren't.
the following are the actual arguments that are passed to the
PythonPlanBinder main method as printed by Arrays.toString():
[C:\Users\Zento\Documents\GitHub\flink\flink-dist\target\flink-1.0-SNAPSHOT-bin\flink-1.0-SNAPSHOT\lib\flink-python_2.10-1.0-SNAPSHOT.jar,
2, resources\python\flink\example\WordCount.py]
Supposedly, these are the exact arguments passed to bin\flink .
i could just ignore the additional argument and have it work again, but
i would still like to know why that argument even arrives here.
On 27.01.2016 18:58, Robert Metzger wrote:
> Given the direction of the path separators, I guess you are using Microsoft
> Windows. I think Fabian is very experienced fixing windows path issues
> before every release ... and usually the issue are paths containing spaces.
> Maybe that's the case here as well?
>
> On Wed, Jan 27, 2016 at 6:51 PM, Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> I was trending more towards changes to the CLI.
>>
>>
>> On 27.01.2016 18:48, Robert Metzger wrote:
>>
>>> Well, according to the git history, it was only me who touched the file
>>> many months ago:
>>>
>>> https://github.com/apache/flink/commits/master/flink-dist/src/main/flink-bin/bin/pyflink2.sh
>>>
>>> On Wed, Jan 27, 2016 at 6:05 PM, Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>> Hello everyone,
>>>> I just noticed an odd behavior that occurs when using the python api
>>>> start
>>>> scripts: i suddenly get another argument i didn't get before; the
>>>> flink-python jar path. which inadvertently breaks everything.
>>>>
>>>> An example as to how the execution of an example should go:
>>>>
>>>> bin\pyflink2.sh resources\python\flink\example\Wordcount.py
>>>> (1 argument)
>>>>
>>>> the pyflink2.sh script then calls:
>>>>
>>>> bin\flink -v libraries\<python-jar> 2(this is the python version)
>>>> resources\python\flink\example\Wordcount.py
>>>> (3 arguments)
>>>>
>>>> which is supposed to pass the arguments "2" and
>>>> "resources\python\flink\example\Wordcount.py" to the PythonPlanBinder's
>>>> main method.
>>>>
>>>> Now for some reason the main method gets 3 arguments, the path to the
>>>> python jar, and the 2 arguments it is supposed to get.
>>>>
>>>> I haven't changed the scripts in a long time, so i assume for now that an
>>>> external change is the cause of this. Did anyone work on something that
>>>> could cause this?
>>>>
>>>>
>>>>
Re: Python start scripts broken
Posted by Robert Metzger <rm...@apache.org>.
Given the direction of the path separators, I guess you are using Microsoft
Windows. I think Fabian is very experienced fixing windows path issues
before every release ... and usually the issue are paths containing spaces.
Maybe that's the case here as well?
On Wed, Jan 27, 2016 at 6:51 PM, Chesnay Schepler <ch...@apache.org>
wrote:
> I was trending more towards changes to the CLI.
>
>
> On 27.01.2016 18:48, Robert Metzger wrote:
>
>> Well, according to the git history, it was only me who touched the file
>> many months ago:
>>
>> https://github.com/apache/flink/commits/master/flink-dist/src/main/flink-bin/bin/pyflink2.sh
>>
>> On Wed, Jan 27, 2016 at 6:05 PM, Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>> Hello everyone,
>>>
>>> I just noticed an odd behavior that occurs when using the python api
>>> start
>>> scripts: i suddenly get another argument i didn't get before; the
>>> flink-python jar path. which inadvertently breaks everything.
>>>
>>> An example as to how the execution of an example should go:
>>>
>>> bin\pyflink2.sh resources\python\flink\example\Wordcount.py
>>> (1 argument)
>>>
>>> the pyflink2.sh script then calls:
>>>
>>> bin\flink -v libraries\<python-jar> 2(this is the python version)
>>> resources\python\flink\example\Wordcount.py
>>> (3 arguments)
>>>
>>> which is supposed to pass the arguments "2" and
>>> "resources\python\flink\example\Wordcount.py" to the PythonPlanBinder's
>>> main method.
>>>
>>> Now for some reason the main method gets 3 arguments, the path to the
>>> python jar, and the 2 arguments it is supposed to get.
>>>
>>> I haven't changed the scripts in a long time, so i assume for now that an
>>> external change is the cause of this. Did anyone work on something that
>>> could cause this?
>>>
>>>
>>>
>
Re: Python start scripts broken
Posted by Chesnay Schepler <ch...@apache.org>.
I was trending more towards changes to the CLI.
On 27.01.2016 18:48, Robert Metzger wrote:
> Well, according to the git history, it was only me who touched the file
> many months ago:
> https://github.com/apache/flink/commits/master/flink-dist/src/main/flink-bin/bin/pyflink2.sh
>
> On Wed, Jan 27, 2016 at 6:05 PM, Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> Hello everyone,
>>
>> I just noticed an odd behavior that occurs when using the python api start
>> scripts: i suddenly get another argument i didn't get before; the
>> flink-python jar path. which inadvertently breaks everything.
>>
>> An example as to how the execution of an example should go:
>>
>> bin\pyflink2.sh resources\python\flink\example\Wordcount.py
>> (1 argument)
>>
>> the pyflink2.sh script then calls:
>>
>> bin\flink -v libraries\<python-jar> 2(this is the python version)
>> resources\python\flink\example\Wordcount.py
>> (3 arguments)
>>
>> which is supposed to pass the arguments "2" and
>> "resources\python\flink\example\Wordcount.py" to the PythonPlanBinder's
>> main method.
>>
>> Now for some reason the main method gets 3 arguments, the path to the
>> python jar, and the 2 arguments it is supposed to get.
>>
>> I haven't changed the scripts in a long time, so i assume for now that an
>> external change is the cause of this. Did anyone work on something that
>> could cause this?
>>
>>
Re: Python start scripts broken
Posted by Robert Metzger <rm...@apache.org>.
Well, according to the git history, it was only me who touched the file
many months ago:
https://github.com/apache/flink/commits/master/flink-dist/src/main/flink-bin/bin/pyflink2.sh
On Wed, Jan 27, 2016 at 6:05 PM, Chesnay Schepler <ch...@apache.org>
wrote:
> Hello everyone,
>
> I just noticed an odd behavior that occurs when using the python api start
> scripts: i suddenly get another argument i didn't get before; the
> flink-python jar path. which inadvertently breaks everything.
>
> An example as to how the execution of an example should go:
>
> bin\pyflink2.sh resources\python\flink\example\Wordcount.py
> (1 argument)
>
> the pyflink2.sh script then calls:
>
> bin\flink -v libraries\<python-jar> 2(this is the python version)
> resources\python\flink\example\Wordcount.py
> (3 arguments)
>
> which is supposed to pass the arguments "2" and
> "resources\python\flink\example\Wordcount.py" to the PythonPlanBinder's
> main method.
>
> Now for some reason the main method gets 3 arguments, the path to the
> python jar, and the 2 arguments it is supposed to get.
>
> I haven't changed the scripts in a long time, so i assume for now that an
> external change is the cause of this. Did anyone work on something that
> could cause this?
>
>