You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/09/27 02:07:00 UTC
[jira] [Commented] (AIRFLOW-668) Configuration parsing doesn't work
properly with python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629663#comment-16629663 ]
ASF GitHub Bot commented on AIRFLOW-668:
----------------------------------------
stale[bot] closed pull request #1918: [AIRFLOW-668] Fix TypeError of run_command on python3
URL: https://github.com/apache/incubator-airflow/pull/1918
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Configuration parsing doesn't work properly with python 3
> ---------------------------------------------------------
>
> Key: AIRFLOW-668
> URL: https://issues.apache.org/jira/browse/AIRFLOW-668
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and CeleryExecutor
> - Python Version: 3.4.5
> Reporter: Rafael Gomes Fernandes
> Assignee: Rafael Gomes Fernandes
> Priority: Major
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
> {noformat}
> File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
> "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfg|borderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on airflow/configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
> command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: <class 'str'> and no error.
> When using python 2 output type: <type 'str'> and no error.
> run_command as it is :
> When using python 3 output type: <class 'bytes'> and TypeError.
> When using python 2 output type: <type 'str'> and no error.
> I have tested the change with Travis CI and it passed, but when I tried to run tox or unit test I found this problems:
> {code:title=run_unit_tests.sh|borderStyle=solid}
> ./run_unit_tests.sh -q -c airflow.cfg
> Initializing the DB
> Starting the unit tests with the following nose arguments: -q -c airflow.cfg
> ======================================================================
> ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table: task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters: ('unit_tests',)])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/failure.py", line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/loader.py", line 418, in loadTestsFromName
> addr.filename, addr.module)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py", line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py", line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, in load_module
> return load_package(name, filename)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, in load_package
> return methods.load()
> File "<frozen importlib._bootstrap>", line 1220, in load
> File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
> File "<frozen importlib._bootstrap>", line 1129, in _exec
> File "<frozen importlib._bootstrap>", line 1471, in exec_module
> File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
> File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, in <module>
> from .contrib import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", line 16, in <module>
> from .operators import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init__.py", line 17, in <module>
> from .ssh_execute_operator import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py", line 37, in <module>
> reset()
> File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py", line 33, in reset
> tis.delete()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3155, in delete
> delete_op.exec_()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py", line 1168, in exec_
> self._do_exec()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py", line 1352, in _do_exec
> mapper=self.mapper)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
> bind, close_with_result=True).execute(clause, params or {})
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 945, in execute
> return meth(self, multiparams, params)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
> return connection._execute_clauseelement(self, multiparams, params)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
> compiled_sql, distilled_params
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
> context)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
> exc_info
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
> reraise(type(exception), exception, tb=exc_tb, cause=cause)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 185, in reraise
> raise value.with_traceback(tb)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
> context)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 469, in do_execute
> cursor.execute(statement, parameters)
> nose.proxy.OperationalError: (sqlite3.OperationalError) no such table: task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters: ('unit_tests',)]
> -------------------- >> begin captured logging << --------------------
> root: INFO: Reading the config from /home/varrun/dev/incubator-airflow/airflow.cfg
> --------------------- >> end captured logging << ---------------------
> ----------------------------------------------------------------------
> Ran 1 test in 0.533s
> {code}
> {code:title=run_tox.sh|borderStyle=solid}
> ./run_tox.sh
> python setup.py test --tox-args="-v -e $TOX_ENV"
> gitpython not found: Cannot compute the git version.
> running test
> Searching for zope.deprecation<5.0,>=4.0
> Reading https://pypi.python.org/simple/zope.deprecation/
> Downloading https://pypi.python.org/packages/51/85/b9d2fdcaf38ce3271ad759c7e8f0e9a28f6d900c1ad495085e4ab1de3795/zope.deprecation-4.2.0.tar.gz#md5=5d110c34ab7b313490c1a8533bbdc7a0
> Best match: zope.deprecation 4.2.0
> Processing zope.deprecation-4.2.0.tar.gz
> Writing /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/setup.cfg
> Running zope.deprecation-4.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/egg-dist-tmp-05ix8s5a
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> creating /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-py3.4.egg
> Extracting zope.deprecation-4.2.0-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-py3.4.egg
> Searching for tabulate<0.8.0,>=0.7.5
> Reading https://pypi.python.org/simple/tabulate/
> Downloading https://pypi.python.org/packages/1c/a1/3367581782ce79b727954f7aa5d29e6a439dc2490a9ac0e7ea0a7115435d/tabulate-0.7.7.tar.gz#md5=39a21aaa9c10be0749c545be34552559
> Best match: tabulate 0.7.7
> Processing tabulate-0.7.7.tar.gz
> Writing /tmp/easy_install-lyp061v6/tabulate-0.7.7/setup.cfg
> Running tabulate-0.7.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-lyp061v6/tabulate-0.7.7/egg-dist-tmp-3lpm_una
> zip_safe flag not set; analyzing archive contents...
> Moving tabulate-0.7.7-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/tabulate-0.7.7-py3.4.egg
> Searching for python-nvd3==0.14.2
> Reading https://pypi.python.org/simple/python-nvd3/
> Downloading https://pypi.python.org/packages/30/68/5d5d7c1a46de23f1da3b4a7035ac305b76aba582648d19cd9da89b5bd8f6/python-nvd3-0.14.2.tar.gz#md5=3a9cc3024368f2542048227a90459cfa
> Best match: python-nvd3 0.14.2
> Processing python-nvd3-0.14.2.tar.gz
> Writing /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/setup.cfg
> Running python-nvd3-0.14.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/egg-dist-tmp-tfryzbr3
> file nvd3.py (for module nvd3) not found
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> warning: no previously-included files matching '__pycache__' found under directory '*'
> warning: no previously-included files matching '*.py[co]' found under directory '*'
> file nvd3.py (for module nvd3) not found
> file nvd3.py (for module nvd3) not found
> creating /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.4.egg
> Extracting python_nvd3-0.14.2-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.4.egg
> Searching for psutil<5.0.0,>=4.2.0
> Reading https://pypi.python.org/simple/psutil/
> Downloading https://pypi.python.org/packages/6c/49/0f784a247868e167389f6ac76b8699b2f3d6f4e8e85685dfec43e58d1ed1/psutil-4.4.2.tar.gz#md5=189749a7b00813dffd6dac516a1166ef
> Best match: psutil 4.4.2
> Processing psutil-4.4.2.tar.gz
> Writing /tmp/easy_install-vygvwvy5/psutil-4.4.2/setup.cfg
> Running psutil-4.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vygvwvy5/psutil-4.4.2/egg-dist-tmp-y3hgclau
> warning: no previously-included files matching '*' found under directory 'docs/_build'
> psutil/_psutil_linux.c: In function ‘PyInit__psutil_linux’:
> psutil/_psutil_linux.c:628: warning: unused variable ‘v’
> zip_safe flag not set; analyzing archive contents...
> psutil.tests.__pycache__.__init__.cpython-34: module references __file__
> psutil.tests.__pycache__.test_linux.cpython-34: module references __file__
> psutil.tests.__pycache__.test_misc.cpython-34: module references __file__
> psutil.tests.__pycache__.test_osx.cpython-34: module references __file__
> psutil.tests.__pycache__.runner.cpython-34: module references __file__
> psutil.tests.__pycache__.test_sunos.cpython-34: module references __file__
> psutil.tests.__pycache__.test_system.cpython-34: module references __file__
> psutil.tests.__pycache__.test_posix.cpython-34: module references __file__
> psutil.tests.__pycache__.test_windows.cpython-34: module references __file__
> psutil.tests.__pycache__.test_process.cpython-34: module references __file__
> psutil.tests.__pycache__.test_bsd.cpython-34: module references __file__
> psutil.tests.__pycache__.test_memory_leaks.cpython-34: module references __file__
> psutil.__pycache__.__init__.cpython-34: module references __file__
> psutil.__pycache__._psutil_linux.cpython-34: module references __file__
> psutil.__pycache__._psutil_posix.cpython-34: module references __file__
> creating /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linux-x86_64.egg
> Extracting psutil-4.4.2-py3.4-linux-x86_64.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linux-x86_64.egg
> Searching for lxml<4.0,>=3.6.0
> Reading https://pypi.python.org/simple/lxml/
> Downloading https://pypi.python.org/packages/4f/3f/cf6daac551fc36cddafa1a71ed48ea5fd642e5feabd3a0d83b8c3dfd0cb4/lxml-3.6.4.tar.gz#md5=6dd7314233029d9dab0156e7b1c7830b
> Best match: lxml 3.6.4
> Processing lxml-3.6.4.tar.gz
> Writing /tmp/easy_install-3iiktavg/lxml-3.6.4/setup.cfg
> Running lxml-3.6.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3iiktavg/lxml-3.6.4/egg-dist-tmp-9hdcvjnx
> Building lxml version 3.6.4.
> Building without Cython.
> Using build configuration of libxslt 1.1.26
> Building against libxml2/libxslt in the following directory: /usr/lib64
> creating /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-x86_64.egg
> Extracting lxml-3.6.4-py3.4-linux-x86_64.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-x86_64.egg
> Searching for gitpython>=2.0.2
> Reading https://pypi.python.org/simple/gitpython/
> Downloading https://pypi.python.org/packages/d7/16/e914d345d7d4e988f9196b9719a99220bad6a5dbd636162f9b5cc35f3bd6/GitPython-2.1.0.tar.gz#md5=29b1fcf504d080dc7a5e630957e829d7
> Best match: GitPython 2.1.0
> Processing GitPython-2.1.0.tar.gz
> Writing /tmp/easy_install-bucom7gp/GitPython-2.1.0/setup.cfg
> Running GitPython-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-bucom7gp/GitPython-2.1.0/egg-dist-tmp-_0q9p8l9
> /home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'test_requirements'
> warnings.warn(msg)
> warning: no files found matching 'README'
> warning: no previously-included files matching '__pycache__' found anywhere in distribution
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> creating /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.egg
> Extracting GitPython-2.1.0-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.egg
> Traceback (most recent call last):
> File "setup.py", line 281, in <module>
> do_setup()
> File "setup.py", line 275, in do_setup
> 'extra_clean': CleanCommand,
> File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", line 955, in run_commands
> self.run_command(cmd)
> File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", line 974, in run_command
> cmd_obj.run()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/command/test.py", line 161, in run
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/dist.py", line 361, in fetch_build_eggs
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 851, in resolve
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 1116, in best_match
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 720, in find
> pkg_resources.VersionConflict: (funcsigs 0.4 (/home/varrun/test/env/airflow3/lib/python3.4/site-packages), Requirement.parse('funcsigs<1.1,>=1.0.2'))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)