You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Kengo Seki (Jira)" <ji...@apache.org> on 2020/07/20 02:14:00 UTC
[jira] [Comment Edited] (BIGTOP-3375) Bump Livy to 0.7.0 and build
it with Python3
[ https://issues.apache.org/jira/browse/BIGTOP-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160855#comment-17160855 ]
Kengo Seki edited comment on BIGTOP-3375 at 7/20/20, 2:13 AM:
--------------------------------------------------------------
I confirmed that building Livy 0.7.0 with Python 3 succeeded on Debian 9/10 and Ubuntu 16.04/18.04 by using a patch that rewrites the python executable name in the patch under bigtop-packages/src/common/livy. Thanks for the advice [~cos] [~evans_ye]!
(And to be precise, it may have been my misunderstanding that livy 0.6 doesn't support Python2, because its Python API submodule doesn't seem to be changed so much between 0.6 and 0.7.)
I still have just one problem. On CentOS 7/8 and Fedora 31, that patch causes the following error, though flake8 is installed and running it manually succeeds.
{code:java}
[INFO] --- exec-maven-plugin:1.2.1:exec (python-api verify) @ livy-python-api ---
Traceback (most recent call last):
File "/usr/local/bin/flake8", line 6, in <module>
from flake8.main.cli import main
ModuleNotFoundError: No module named 'flake8'
{code}
I've not figured out the root cause of this failure for now, but this step just runs flake8 for linting, so I think we can skip it.
was (Author: sekikn):
I confirmed that building Livy 0.7.0 with Python 3 succeeded on Debian 9/10 and Ubuntu 16.04/18.04 by using a patch that rewrites the python executable name in the patch under bigtop-packages/src/common/livy. Thanks for the advice [~cos] [~evans_ye]!
(And to be precise, it may have been my misunderstanding that livy 0.6 doesn't support Python2, because its Python API submodule doesn't seem to be changed so much between 0.6 and 0.7.)
I still have just one problem. On CentOS 7/8 and Fedora 31, the attached patch causes the following error, though flake8 is installed and running it manually succeeds.
{code:java}
[INFO] --- exec-maven-plugin:1.2.1:exec (python-api verify) @ livy-python-api ---
Traceback (most recent call last):
File "/usr/local/bin/flake8", line 6, in <module>
from flake8.main.cli import main
ModuleNotFoundError: No module named 'flake8'
{code}
I've not figured out the root cause of this failure for now, but this step just runs flake8 for linting, so I think we can skip it.
> Bump Livy to 0.7.0 and build it with Python3
> --------------------------------------------
>
> Key: BIGTOP-3375
> URL: https://issues.apache.org/jira/browse/BIGTOP-3375
> Project: Bigtop
> Issue Type: Bug
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Major
>
> We changed the way to install flake8 and setuptools to use Python3 instead of Python2 in BIGTOP-3364, but I just noticed Livy 0.6.0 only supports Python 2.x, so running {{./gradlew livy-pkg-ind}} fails as follows.
> {code:java}
> $ ./gradlew livy-clean livy-pkg-ind
> (snip)
> Installed /bigtop/build/livy/rpm/BUILD/livy-0.6.0/python-api/UNKNOWN-0.0.0-py2.7.egg
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> no previously-included directories found matching 'docs/build/'
> warning: install_lib: 'build/lib' does not exist -- no Python modules to install
> zip_safe flag not set; analyzing archive contents...
> Traceback (most recent call last):
> File "setup.py", line 57, in <module>
> tests_require=['pytest']
> File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
> parse_requirements(requires), installer=self.fetch_build_egg
> File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
> raise DistributionNotFound(req)
> pkg_resources.DistributionNotFound: flake8
> (snip)
> [INFO] livy-examples ...................................... SUCCESS [ 9.374 s]
> [INFO] livy-python-api 0.6.0-incubating ................... FAILURE [ 2.545 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 10:10 min
> [INFO] Finished at: 2020-07-08T23:20:10Z
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> {code}
> I tried to install the python-flake8 package to avoid this, but I got another error:
> {code:java}
> [INFO] --- exec-maven-plugin:1.2.1:exec (python-api install) @ livy-python-api ---
> Traceback (most recent call last):
> File "setup.py", line 57, in <module>
> tests_require=['pytest']
> File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 265, in __init__
> self.fetch_build_eggs(attrs.pop('setup_requires'))
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
> parse_requirements(requires), installer=self.fetch_build_egg
> File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 618, in resolve
> dist = best[req.key] = env.best_match(req, self, installer)
> File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 862, in best_match
> return self.obtain(req, installer) # try and download/install
> File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 874, in obtain
> return installer(requirement)
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 339, in fetch_build_egg
> return cmd.easy_install(req)
> File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
> return self.install_item(spec, dist.location, tmpdir, deps)
> File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 653, in install_item
> dists = self.install_eggs(spec, download, tmpdir)
> File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
> return self.build_and_install(setup_script, setup_base)
> File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
> self.run_setup(setup_script, setup_base, args)
> File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
> run_setup(setup_script, args)
> File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 69, in run_setup
> lambda: execfile(
> File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 120, in run
> return func()
> File "/usr/lib/python2.7/site-packages/setuptools/sandbox.py", line 71, in <lambda>
> {'__file__':setup_script, '__name__':'__main__'}
> File "setup.py", line 21, in <module>
>
> File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
> _setup_distribution = dist = klass(attrs)
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 269, in __init__
> _Distribution.__init__(self,attrs)
> File "/usr/lib64/python2.7/distutils/dist.py", line 287, in __init__
> self.finalize_options()
> File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 302, in finalize_options
> ep.load()(self, ep.name, value)
> File "build/bdist.linux-x86_64/egg/setuptools_scm/integration.py", line 9, in version_keyword
> File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 60, in _warn_if_setuptools_outdated
> setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
> (snip)
> [INFO] livy-examples ...................................... SUCCESS [ 4.536 s]
> [INFO] livy-python-api 0.6.0-incubating ................... FAILURE [ 0.887 s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:21 min
> [INFO] Finished at: 2020-07-09T22:57:06Z
> [INFO] ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-api install) on project livy-python-api: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
> {code}
> CentOS 7 provides the python-setuptools package, but its version is 0.9.8. So we may have to install setuptools as a Python2 library via pip2, as we did before.
> As Yuqi mentioned before, the latest version of setuptools has already dropped Python2 support, but prior to v45.0.0 seems to work with Python2.
> [https://setuptools.readthedocs.io/en/latest/history.html#v45-0-0]
> So I'll try to revert BIGTOP-3364 and pin setuptools' version to 44.0.0, and check if both Zeppelin and Livy are successfully built on all distros.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)