You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Rafael Gomes Fernandes (JIRA)" <ji...@apache.org> on 2016/12/02 19:45:58 UTC
[jira] [Updated] (AIRFLOW-668) Configuration parsing doesn't work
properly with python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rafael Gomes Fernandes updated AIRFLOW-668:
-------------------------------------------
Description:
The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
{noformat}
File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
TypeError: 'str' does not support the buffer interface
{noformat}
To reproduce the problem change the following line on airflow.cfg:
{code:title=airflow.cfgborderStyle=solid}
sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
{code}
The solution is change the following run_command method's line on configuration.py:
{code:title=airflow/configuration.py|borderStyle=solid}
def run_command(command):
"""
Runs command and returns stdout
"""
process = subprocess.Popen(
command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE{color:green}, universal_newlines=True{color})
output, stderr = process.communicate()
if process.returncode != 0:
raise AirflowConfigException(
"Cannot execute {}. Error code is: {}. Output: {}, Stderr: {}"
.format(command, process.returncode, output, stderr)
)
return output
{code}
By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.
run_command with universal_newlines=True:
When using python 3 output type: <class 'str'> and no error.
When using python 2 output type: <type 'str'> and no error.
run_command as it is :
When using python 3 output type: <class 'bytes'> and TypeError.
When using python 2 output type: <type 'str'> and no error.
was:
The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
"sqlite" in self.get('core', 'sql_alchemy_conn')):
TypeError: 'str' does not support the buffer interface
To reproduce the problem change the following line on airflow.cfg:
sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
The solution is change the following run_command method's line on configuration.py:
command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.
run_command with universal_newlines=True:
When using python 3 output type: <class 'str'> and no error.
When using python 2 output type: <type 'str'> and no error.
run_command as it is :
When using python 3 output type: <class 'bytes'> and TypeError.
When using python 2 output type: <type 'str'> and no error.
> Configuration parsing doesn't work properly with python 3
> ---------------------------------------------------------
>
> Key: AIRFLOW-668
> URL: https://issues.apache.org/jira/browse/AIRFLOW-668
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Airflow version: v1.7.1.3
> - Airflow components: webserver and scheduler with a postgres database and CeleryExecutor
> - Python Version: 3.4.5
> Reporter: Rafael Gomes Fernandes
> Assignee: Rafael Gomes Fernandes
>
> The problem is: if you use python3 and the '_cmd' on the config file airflow will not start due the error:
> {noformat}
> File "~/test/env/airflow3/lib/python3.4/site-packages/airflow/configuration.py", line 447, in _validate
> "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: 'str' does not support the buffer interface
> {noformat}
> To reproduce the problem change the following line on airflow.cfg:
> {code:title=airflow.cfgborderStyle=solid}
> sql_alchemy_conn_cmd = echo sqlite:////~/airflow/airflow.db
> {code}
> The solution is change the following run_command method's line on configuration.py:
> {code:title=airflow/configuration.py|borderStyle=solid}
> def run_command(command):
> """
> Runs command and returns stdout
> """
> process = subprocess.Popen(
> command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE{color:green}, universal_newlines=True{color})
> output, stderr = process.communicate()
> if process.returncode != 0:
> raise AirflowConfigException(
> "Cannot execute {}. Error code is: {}. Output: {}, Stderr: {}"
> .format(command, process.returncode, output, stderr)
> )
> return output
> {code}
> By setting the universal_newlines to true the file objects stdout and stderr are opened as text files and treated as string in python 2 and python 3 avoiding the error.
> run_command with universal_newlines=True:
> When using python 3 output type: <class 'str'> and no error.
> When using python 2 output type: <type 'str'> and no error.
> run_command as it is :
> When using python 3 output type: <class 'bytes'> and TypeError.
> When using python 2 output type: <type 'str'> and no error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)