You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Alec Swan <al...@gmail.com> on 2016/05/03 00:34:59 UTC
Configuring Python environment
Hello,
I am having a real hard time configuring Storm to use python2.7 virtual
environment that I installed on my CentOS 6.5 host.
CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
Switching to python2.7 breaks some of CentOS functionality, so I had to
install python2.7 as a separate environment.
I also read that if I set BASH_ENV env var to point to the script that
activates the virtual environment, then it will be loaded before scripts
are executed. I also changed the first line in storm.py to:
#!/usr/bin/env python
so that the right python executable is being used. However, I still have no
luck and keep getting the error shown below after topology is successfully
submitted.
Has anyone figured out how to use Storm with python virtual environments?
2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess
python: error while loading shared libraries: libpython2.7.so.1.0: cannot
open shared object file: No such file or directory
at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
Caused by: java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method) ~[?:1.8.0_73]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[?:1.8.0_73]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_73]
at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_73]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[?:1.8.0_73]
at
backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
~[storm-core-0.10.0.jar:0.10.0]
... 5 more
Thanks,
Alec
Re: Configuring Python environment
Posted by Alec Swan <al...@gmail.com>.
I did try streamparse, but it failed to deploy the topology to the remote
Storm cluster because of some internal errors which I wasn't able to track
down. I am not surprised though, because even manual install of python 2.7
or virtual environment is painful on CentOS 6.5.
So, I took the topology JAR produced by Streamparse and tried to manually
deploy on Storm, but ran into the Python issues that I am trying to resolve
in this thread.
Thanks,
Alec
On Mon, May 2, 2016 at 8:40 PM, Jiaming Lin <li...@gmail.com> wrote:
> You can try streamparse, I haven't used it, but it looks charming.
>
>
> On Tuesday, May 3, 2016, Alec Swan <al...@gmail.com> wrote:
>
>> Thanks, Joaquin, but I've seen all these articles since I spent the
>> entire weekend trying to get my topology to run on CentOS/Python2.7.
>> Getting it to work on Mac OSX was a breeze because I can easily install
>> Python 2.7 on it.
>>
>> It'd be great if somebody with the actually CentOS/Python2.7 experience
>> could share their thoughts.
>>
>> Thanks,
>>
>> Alec
>>
>> On Mon, May 2, 2016 at 7:59 PM, Joaquin Menchaca <jm...@gobalto.com>
>> wrote:
>>
>>> virtualenvwrappers
>>> - http://virtualenvwrapper.readthedocs.io/en/latest/
>>> - https://pypi.python.org/pypi/virtualenvwrapper/
>>> Pyenv
>>> - https://amaral.northwestern.edu/resources/guides/pyenv-tutorial
>>> - https://github.com/yyuu/pyenv
>>> Docker
>>>
>>> You can have a Python in another directory, then put that directory in
>>> the path. Maybe linuxbrew can help with that.
>>>
>>> Linuxbrew - http://linuxbrew.sh
>>>
>>> Disclaimer, I haven't tested any of these with Storm. I personally
>>> don't use CentOS (or RHEL) any more, because they are dinosaurs when it
>>> comes to scripting languages and web development. But many love the slow
>>> churn and stability.
>>>
>>> I also found these:
>>> -
>>> https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only
>>> -
>>> https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
>>> - http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
>>> On May 2, 2016 3:35 PM, "Alec Swan" <al...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am having a real hard time configuring Storm to use python2.7 virtual
>>>> environment that I installed on my CentOS 6.5 host.
>>>>
>>>> CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
>>>> Switching to python2.7 breaks some of CentOS functionality, so I had to
>>>> install python2.7 as a separate environment.
>>>>
>>>> I also read that if I set BASH_ENV env var to point to the script that
>>>> activates the virtual environment, then it will be loaded before scripts
>>>> are executed. I also changed the first line in storm.py to:
>>>> #!/usr/bin/env python
>>>> so that the right python executable is being used. However, I still
>>>> have no luck and keep getting the error shown below after topology is
>>>> successfully submitted.
>>>>
>>>> Has anyone figured out how to use Storm with python virtual
>>>> environments?
>>>>
>>>> 2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
>>>> java.lang.RuntimeException: Error when launching multilang subprocess
>>>> python: error while loading shared libraries: libpython2.7.so.1.0:
>>>> cannot open shared object file: No such file or directory
>>>>
>>>> at
>>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> at
>>>> backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
>>>> [storm-core-0.10.0.jar:0.10.0]
>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
>>>> Caused by: java.io.IOException: Broken pipe
>>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>>> ~[?:1.8.0_73]
>>>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>>> ~[?:1.8.0_73]
>>>> at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>> ~[?:1.8.0_73]
>>>> at
>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>> ~[?:1.8.0_73]
>>>> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
>>>> ~[?:1.8.0_73]
>>>> at
>>>> backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> at
>>>> backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> at
>>>> backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> at
>>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>> ... 5 more
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Alec
>>>>
>>>
>>
Re: Configuring Python environment
Posted by Jiaming Lin <li...@gmail.com>.
You can try streamparse, I haven't used it, but it looks charming.
On Tuesday, May 3, 2016, Alec Swan <al...@gmail.com> wrote:
> Thanks, Joaquin, but I've seen all these articles since I spent the entire
> weekend trying to get my topology to run on CentOS/Python2.7.
> Getting it to work on Mac OSX was a breeze because I can easily install
> Python 2.7 on it.
>
> It'd be great if somebody with the actually CentOS/Python2.7 experience
> could share their thoughts.
>
> Thanks,
>
> Alec
>
> On Mon, May 2, 2016 at 7:59 PM, Joaquin Menchaca <jmenchaca@gobalto.com
> <javascript:_e(%7B%7D,'cvml','jmenchaca@gobalto.com');>> wrote:
>
>> virtualenvwrappers
>> - http://virtualenvwrapper.readthedocs.io/en/latest/
>> - https://pypi.python.org/pypi/virtualenvwrapper/
>> Pyenv
>> - https://amaral.northwestern.edu/resources/guides/pyenv-tutorial
>> - https://github.com/yyuu/pyenv
>> Docker
>>
>> You can have a Python in another directory, then put that directory in
>> the path. Maybe linuxbrew can help with that.
>>
>> Linuxbrew - http://linuxbrew.sh
>>
>> Disclaimer, I haven't tested any of these with Storm. I personally don't
>> use CentOS (or RHEL) any more, because they are dinosaurs when it comes to
>> scripting languages and web development. But many love the slow churn and
>> stability.
>>
>> I also found these:
>> -
>> https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only
>> -
>> https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
>> - http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
>> On May 2, 2016 3:35 PM, "Alec Swan" <alecswan@gmail.com
>> <javascript:_e(%7B%7D,'cvml','alecswan@gmail.com');>> wrote:
>>
>>> Hello,
>>>
>>> I am having a real hard time configuring Storm to use python2.7 virtual
>>> environment that I installed on my CentOS 6.5 host.
>>>
>>> CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
>>> Switching to python2.7 breaks some of CentOS functionality, so I had to
>>> install python2.7 as a separate environment.
>>>
>>> I also read that if I set BASH_ENV env var to point to the script that
>>> activates the virtual environment, then it will be loaded before scripts
>>> are executed. I also changed the first line in storm.py to:
>>> #!/usr/bin/env python
>>> so that the right python executable is being used. However, I still have
>>> no luck and keep getting the error shown below after topology is
>>> successfully submitted.
>>>
>>> Has anyone figured out how to use Storm with python virtual environments?
>>>
>>> 2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
>>> java.lang.RuntimeException: Error when launching multilang subprocess
>>> python: error while loading shared libraries: libpython2.7.so.1.0:
>>> cannot open shared object file: No such file or directory
>>>
>>> at
>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
>>> [storm-core-0.10.0.jar:0.10.0]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
>>> Caused by: java.io.IOException: Broken pipe
>>> at java.io.FileOutputStream.writeBytes(Native Method)
>>> ~[?:1.8.0_73]
>>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>> ~[?:1.8.0_73]
>>> at
>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>> ~[?:1.8.0_73]
>>> at
>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>> ~[?:1.8.0_73]
>>> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
>>> ~[?:1.8.0_73]
>>> at
>>> backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> ... 5 more
>>>
>>>
>>> Thanks,
>>>
>>> Alec
>>>
>>
>
Re: Configuring Python environment
Posted by Alec Swan <al...@gmail.com>.
Thanks, Joaquin, but I've seen all these articles since I spent the entire
weekend trying to get my topology to run on CentOS/Python2.7.
Getting it to work on Mac OSX was a breeze because I can easily install
Python 2.7 on it.
It'd be great if somebody with the actually CentOS/Python2.7 experience
could share their thoughts.
Thanks,
Alec
On Mon, May 2, 2016 at 7:59 PM, Joaquin Menchaca <jm...@gobalto.com>
wrote:
> virtualenvwrappers
> - http://virtualenvwrapper.readthedocs.io/en/latest/
> - https://pypi.python.org/pypi/virtualenvwrapper/
> Pyenv
> - https://amaral.northwestern.edu/resources/guides/pyenv-tutorial
> - https://github.com/yyuu/pyenv
> Docker
>
> You can have a Python in another directory, then put that directory in the
> path. Maybe linuxbrew can help with that.
>
> Linuxbrew - http://linuxbrew.sh
>
> Disclaimer, I haven't tested any of these with Storm. I personally don't
> use CentOS (or RHEL) any more, because they are dinosaurs when it comes to
> scripting languages and web development. But many love the slow churn and
> stability.
>
> I also found these:
> -
> https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only
> -
> https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
> - http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
> On May 2, 2016 3:35 PM, "Alec Swan" <al...@gmail.com> wrote:
>
>> Hello,
>>
>> I am having a real hard time configuring Storm to use python2.7 virtual
>> environment that I installed on my CentOS 6.5 host.
>>
>> CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
>> Switching to python2.7 breaks some of CentOS functionality, so I had to
>> install python2.7 as a separate environment.
>>
>> I also read that if I set BASH_ENV env var to point to the script that
>> activates the virtual environment, then it will be loaded before scripts
>> are executed. I also changed the first line in storm.py to:
>> #!/usr/bin/env python
>> so that the right python executable is being used. However, I still have
>> no luck and keep getting the error shown below after topology is
>> successfully submitted.
>>
>> Has anyone figured out how to use Storm with python virtual environments?
>>
>> 2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
>> java.lang.RuntimeException: Error when launching multilang subprocess
>> python: error while loading shared libraries: libpython2.7.so.1.0: cannot
>> open shared object file: No such file or directory
>>
>> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
>> [storm-core-0.10.0.jar:0.10.0]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
>> Caused by: java.io.IOException: Broken pipe
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> ~[?:1.8.0_73]
>> at java.io.FileOutputStream.write(FileOutputStream.java:326)
>> ~[?:1.8.0_73]
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> ~[?:1.8.0_73]
>> at
>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>> ~[?:1.8.0_73]
>> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
>> ~[?:1.8.0_73]
>> at
>> backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> ... 5 more
>>
>>
>> Thanks,
>>
>> Alec
>>
>
Re: Configuring Python environment
Posted by Joaquin Menchaca <jm...@gobalto.com>.
virtualenvwrappers
- http://virtualenvwrapper.readthedocs.io/en/latest/
- https://pypi.python.org/pypi/virtualenvwrapper/
Pyenv
- https://amaral.northwestern.edu/resources/guides/pyenv-tutorial
- https://github.com/yyuu/pyenv
Docker
You can have a Python in another directory, then put that directory in the
path. Maybe linuxbrew can help with that.
Linuxbrew - http://linuxbrew.sh
Disclaimer, I haven't tested any of these with Storm. I personally don't
use CentOS (or RHEL) any more, because they are dinosaurs when it comes to
scripting languages and web development. But many love the slow churn and
stability.
I also found these:
-
https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only
-
https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
- http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
On May 2, 2016 3:35 PM, "Alec Swan" <al...@gmail.com> wrote:
> Hello,
>
> I am having a real hard time configuring Storm to use python2.7 virtual
> environment that I installed on my CentOS 6.5 host.
>
> CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
> Switching to python2.7 breaks some of CentOS functionality, so I had to
> install python2.7 as a separate environment.
>
> I also read that if I set BASH_ENV env var to point to the script that
> activates the virtual environment, then it will be loaded before scripts
> are executed. I also changed the first line in storm.py to:
> #!/usr/bin/env python
> so that the right python executable is being used. However, I still have
> no luck and keep getting the error shown below after topology is
> successfully submitted.
>
> Has anyone figured out how to use Storm with python virtual environments?
>
> 2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: Error when launching multilang subprocess
> python: error while loading shared libraries: libpython2.7.so.1.0: cannot
> open shared object file: No such file or directory
>
> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
> ~[storm-core-0.10.0.jar:0.10.0]
> at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
> ~[storm-core-0.10.0.jar:0.10.0]
> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
> [storm-core-0.10.0.jar:0.10.0]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method) ~[?:1.8.0_73]
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> ~[?:1.8.0_73]
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[?:1.8.0_73]
> at
> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[?:1.8.0_73]
> at java.io.DataOutputStream.flush(DataOutputStream.java:123)
> ~[?:1.8.0_73]
> at
> backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
> ~[storm-core-0.10.0.jar:0.10.0]
> at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
> ~[storm-core-0.10.0.jar:0.10.0]
> ... 5 more
>
>
> Thanks,
>
> Alec
>