You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ash Berlin-Taylor (JIRA)" <ji...@apache.org> on 2018/09/27 09:14:00 UTC
[jira] [Resolved] (AIRFLOW-668) Configuration parsing doesn't work
properly with python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ash Berlin-Taylor resolved AIRFLOW-668.
---------------------------------------
Resolution: Fixed
No longer an issue.
Tested with this config on Python3.5.2
{code}
sql_alchemy_conn_cmd = echo sqlite:////Users/ash/airflow/airflow-2.db
{code}
> Configuration parsing doesn't work properly with python 3
> ---------------------------------------------------------
>
> Key: AIRFLOW-668
> URL: https://issues.apache.org/jira/browse/AIRFLOW-668
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and CeleryExecutor
> - Python Version: 3.4.5
> Reporter: Rafael Gomes Fernandes
> Assignee: Rafael Gomes Fernandes
> Priority: Major
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
> {noformat}
> File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
> "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfg|borderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on airflow/configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
> command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: <class 'str'> and no error.
> When using python 2 output type: <type 'str'> and no error.
> run_command as it is :
> When using python 3 output type: <class 'bytes'> and TypeError.
> When using python 2 output type: <type 'str'> and no error.
> I have tested the change with Travis CI and it passed, but when I tried to run tox or unit test I found this problems:
> {code:title=run_unit_tests.sh|borderStyle=solid}
> ./run_unit_tests.sh -q -c airflow.cfg
> Initializing the DB
> Starting the unit tests with the following nose arguments: -q -c airflow.cfg
> ======================================================================
> ERROR: Failure: OperationalError ((sqlite3.OperationalError) no such table: task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters: ('unit_tests',)])
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/failure.py", line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/loader.py", line 418, in loadTestsFromName
> addr.filename, addr.module)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py", line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/nose/importer.py", line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 245, in load_module
> return load_package(name, filename)
> File "/home/varrun/test/env/airflow3/lib/python3.4/imp.py", line 217, in load_package
> return methods.load()
> File "<frozen importlib._bootstrap>", line 1220, in load
> File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
> File "<frozen importlib._bootstrap>", line 1129, in _exec
> File "<frozen importlib._bootstrap>", line 1471, in exec_module
> File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
> File "/home/varrun/dev/incubator-airflow/tests/__init__.py", line 18, in <module>
> from .contrib import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/__init__.py", line 16, in <module>
> from .operators import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/__init__.py", line 17, in <module>
> from .ssh_execute_operator import *
> File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py", line 37, in <module>
> reset()
> File "/home/varrun/dev/incubator-airflow/tests/contrib/operators/ssh_execute_operator.py", line 33, in reset
> tis.delete()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3155, in delete
> delete_op.exec_()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py", line 1168, in exec_
> self._do_exec()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/persistence.py", line 1352, in _do_exec
> mapper=self.mapper)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
> bind, close_with_result=True).execute(clause, params or {})
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 945, in execute
> return meth(self, multiparams, params)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
> return connection._execute_clauseelement(self, multiparams, params)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
> compiled_sql, distilled_params
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
> context)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
> exc_info
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
> reraise(type(exception), exception, tb=exc_tb, cause=cause)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 185, in reraise
> raise value.with_traceback(tb)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
> context)
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 469, in do_execute
> cursor.execute(statement, parameters)
> nose.proxy.OperationalError: (sqlite3.OperationalError) no such table: task_instance [SQL: 'DELETE FROM task_instance WHERE task_instance.dag_id = ?'] [parameters: ('unit_tests',)]
> -------------------- >> begin captured logging << --------------------
> root: INFO: Reading the config from /home/varrun/dev/incubator-airflow/airflow.cfg
> --------------------- >> end captured logging << ---------------------
> ----------------------------------------------------------------------
> Ran 1 test in 0.533s
> {code}
> {code:title=run_tox.sh|borderStyle=solid}
> ./run_tox.sh
> python setup.py test --tox-args="-v -e $TOX_ENV"
> gitpython not found: Cannot compute the git version.
> running test
> Searching for zope.deprecation<5.0,>=4.0
> Reading https://pypi.python.org/simple/zope.deprecation/
> Downloading https://pypi.python.org/packages/51/85/b9d2fdcaf38ce3271ad759c7e8f0e9a28f6d900c1ad495085e4ab1de3795/zope.deprecation-4.2.0.tar.gz#md5=5d110c34ab7b313490c1a8533bbdc7a0
> Best match: zope.deprecation 4.2.0
> Processing zope.deprecation-4.2.0.tar.gz
> Writing /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/setup.cfg
> Running zope.deprecation-4.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_usl2kjj/zope.deprecation-4.2.0/egg-dist-tmp-05ix8s5a
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> creating /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-py3.4.egg
> Extracting zope.deprecation-4.2.0-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/zope.deprecation-4.2.0-py3.4.egg
> Searching for tabulate<0.8.0,>=0.7.5
> Reading https://pypi.python.org/simple/tabulate/
> Downloading https://pypi.python.org/packages/1c/a1/3367581782ce79b727954f7aa5d29e6a439dc2490a9ac0e7ea0a7115435d/tabulate-0.7.7.tar.gz#md5=39a21aaa9c10be0749c545be34552559
> Best match: tabulate 0.7.7
> Processing tabulate-0.7.7.tar.gz
> Writing /tmp/easy_install-lyp061v6/tabulate-0.7.7/setup.cfg
> Running tabulate-0.7.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-lyp061v6/tabulate-0.7.7/egg-dist-tmp-3lpm_una
> zip_safe flag not set; analyzing archive contents...
> Moving tabulate-0.7.7-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/tabulate-0.7.7-py3.4.egg
> Searching for python-nvd3==0.14.2
> Reading https://pypi.python.org/simple/python-nvd3/
> Downloading https://pypi.python.org/packages/30/68/5d5d7c1a46de23f1da3b4a7035ac305b76aba582648d19cd9da89b5bd8f6/python-nvd3-0.14.2.tar.gz#md5=3a9cc3024368f2542048227a90459cfa
> Best match: python-nvd3 0.14.2
> Processing python-nvd3-0.14.2.tar.gz
> Writing /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/setup.cfg
> Running python-nvd3-0.14.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-kjjo6ak_/python-nvd3-0.14.2/egg-dist-tmp-tfryzbr3
> file nvd3.py (for module nvd3) not found
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> warning: no previously-included files matching '__pycache__' found under directory '*'
> warning: no previously-included files matching '*.py[co]' found under directory '*'
> file nvd3.py (for module nvd3) not found
> file nvd3.py (for module nvd3) not found
> creating /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.4.egg
> Extracting python_nvd3-0.14.2-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/python_nvd3-0.14.2-py3.4.egg
> Searching for psutil<5.0.0,>=4.2.0
> Reading https://pypi.python.org/simple/psutil/
> Downloading https://pypi.python.org/packages/6c/49/0f784a247868e167389f6ac76b8699b2f3d6f4e8e85685dfec43e58d1ed1/psutil-4.4.2.tar.gz#md5=189749a7b00813dffd6dac516a1166ef
> Best match: psutil 4.4.2
> Processing psutil-4.4.2.tar.gz
> Writing /tmp/easy_install-vygvwvy5/psutil-4.4.2/setup.cfg
> Running psutil-4.4.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vygvwvy5/psutil-4.4.2/egg-dist-tmp-y3hgclau
> warning: no previously-included files matching '*' found under directory 'docs/_build'
> psutil/_psutil_linux.c: In function ‘PyInit__psutil_linux’:
> psutil/_psutil_linux.c:628: warning: unused variable ‘v’
> zip_safe flag not set; analyzing archive contents...
> psutil.tests.__pycache__.__init__.cpython-34: module references __file__
> psutil.tests.__pycache__.test_linux.cpython-34: module references __file__
> psutil.tests.__pycache__.test_misc.cpython-34: module references __file__
> psutil.tests.__pycache__.test_osx.cpython-34: module references __file__
> psutil.tests.__pycache__.runner.cpython-34: module references __file__
> psutil.tests.__pycache__.test_sunos.cpython-34: module references __file__
> psutil.tests.__pycache__.test_system.cpython-34: module references __file__
> psutil.tests.__pycache__.test_posix.cpython-34: module references __file__
> psutil.tests.__pycache__.test_windows.cpython-34: module references __file__
> psutil.tests.__pycache__.test_process.cpython-34: module references __file__
> psutil.tests.__pycache__.test_bsd.cpython-34: module references __file__
> psutil.tests.__pycache__.test_memory_leaks.cpython-34: module references __file__
> psutil.__pycache__.__init__.cpython-34: module references __file__
> psutil.__pycache__._psutil_linux.cpython-34: module references __file__
> psutil.__pycache__._psutil_posix.cpython-34: module references __file__
> creating /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linux-x86_64.egg
> Extracting psutil-4.4.2-py3.4-linux-x86_64.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/psutil-4.4.2-py3.4-linux-x86_64.egg
> Searching for lxml<4.0,>=3.6.0
> Reading https://pypi.python.org/simple/lxml/
> Downloading https://pypi.python.org/packages/4f/3f/cf6daac551fc36cddafa1a71ed48ea5fd642e5feabd3a0d83b8c3dfd0cb4/lxml-3.6.4.tar.gz#md5=6dd7314233029d9dab0156e7b1c7830b
> Best match: lxml 3.6.4
> Processing lxml-3.6.4.tar.gz
> Writing /tmp/easy_install-3iiktavg/lxml-3.6.4/setup.cfg
> Running lxml-3.6.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3iiktavg/lxml-3.6.4/egg-dist-tmp-9hdcvjnx
> Building lxml version 3.6.4.
> Building without Cython.
> Using build configuration of libxslt 1.1.26
> Building against libxml2/libxslt in the following directory: /usr/lib64
> creating /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-x86_64.egg
> Extracting lxml-3.6.4-py3.4-linux-x86_64.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/lxml-3.6.4-py3.4-linux-x86_64.egg
> Searching for gitpython>=2.0.2
> Reading https://pypi.python.org/simple/gitpython/
> Downloading https://pypi.python.org/packages/d7/16/e914d345d7d4e988f9196b9719a99220bad6a5dbd636162f9b5cc35f3bd6/GitPython-2.1.0.tar.gz#md5=29b1fcf504d080dc7a5e630957e829d7
> Best match: GitPython 2.1.0
> Processing GitPython-2.1.0.tar.gz
> Writing /tmp/easy_install-bucom7gp/GitPython-2.1.0/setup.cfg
> Running GitPython-2.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-bucom7gp/GitPython-2.1.0/egg-dist-tmp-_0q9p8l9
> /home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'test_requirements'
> warnings.warn(msg)
> warning: no files found matching 'README'
> warning: no previously-included files matching '__pycache__' found anywhere in distribution
> warning: no previously-included files matching '*.pyc' found anywhere in distribution
> creating /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.egg
> Extracting GitPython-2.1.0-py3.4.egg to /home/varrun/dev/incubator-airflow/.eggs
> Installed /home/varrun/dev/incubator-airflow/.eggs/GitPython-2.1.0-py3.4.egg
> Traceback (most recent call last):
> File "setup.py", line 281, in <module>
> do_setup()
> File "setup.py", line 275, in do_setup
> 'extra_clean': CleanCommand,
> File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/core.py", line 148, in setup
> dist.run_commands()
> File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", line 955, in run_commands
> self.run_command(cmd)
> File "/home/varrun/test/env/airflow3/lib/python3.4/distutils/dist.py", line 974, in run_command
> cmd_obj.run()
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/command/test.py", line 161, in run
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/setuptools/dist.py", line 361, in fetch_build_eggs
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 851, in resolve
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 1116, in best_match
> File "/home/varrun/test/env/airflow3/lib/python3.4/site-packages/setuptools-27.2.0-py3.4.egg/pkg_resources/__init__.py", line 720, in find
> pkg_resources.VersionConflict: (funcsigs 0.4 (/home/varrun/test/env/airflow3/lib/python3.4/site-packages), Requirement.parse('funcsigs<1.1,>=1.0.2'))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)