You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Nimrod Morag (JIRA)" <ji...@apache.org> on 2018/11/25 08:29:00 UTC
[jira] [Updated] (AIRFLOW-3394) Cannot redirect output from
subprocesses
[ https://issues.apache.org/jira/browse/AIRFLOW-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nimrod Morag updated AIRFLOW-3394:
----------------------------------
Description:
When using PythonOperator, I'm using the subprocess module and redirecting the output to sys.stdout in the parent process
since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object has no attribute 'fileno' Traceback (most recent call last):}}
{{File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in _run_raw_task result = task_copy.execute(context=context)_}}
{{_File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch = super(MyMultiBranchPythonOperator, self).execute(context)_}}
{{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 95, in execute return_value = self.execute_callable()_}}
{{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 100, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs)_}}
{{_File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config')_}}
{{_File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True):_}}
{{_File "/usr/lib/python3.5/subprocess.py", line 693, in run with Popen(*popenargs, **kwargs) as process:_}}
{{_File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite) = self._get_handles(stdin, stdout, stderr)}}
{{File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite = stdout.fileno()}}
{{AttributeError: 'StreamLogWriter' object has no attribute 'fileno'}}
{{[2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}
{{Fixed locally by adding to the StreamLogWriter class:}}
{{def fileno(self):}}
{{ return 1}}
{{(1 is stdout's file descriptor)}}
was:
When using PythonOperator, I'm using the subprocess module and redirecting the output to sys.stdout in the parent process
since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object has no attribute 'fileno' Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in _run_raw_task result = task_copy.execute(context=context) File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch = super(MyMultiBranchPythonOperator, self).execute(context) File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 95, in execute return_value = self.execute_callable() File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 100, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs) File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config') File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True): File "/usr/lib/python3.5/subprocess.py", line 693, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite) = self._get_handles(stdin, stdout, stderr) File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite = stdout.fileno() AttributeError: 'StreamLogWriter' object has no attribute 'fileno' [2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}
{{Fixed locally by adding to the StreamLogWriter class:}}
{{def fileno(self):}}
{{ return 1}}
{{(1 is stdout's file descriptor)}}
> Cannot redirect output from subprocesses
> ----------------------------------------
>
> Key: AIRFLOW-3394
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3394
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
> Affects Versions: 1.10.1
> Environment: ubuntu 16.04
> python 3.5.2
> Reporter: Nimrod Morag
> Priority: Major
> Labels: beginner, easyfix, patch, usability
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> When using PythonOperator, I'm using the subprocess module and redirecting the output to sys.stdout in the parent process
>
> since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
> {{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object has no attribute 'fileno' Traceback (most recent call last):}}
> {{File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in _run_raw_task result = task_copy.execute(context=context)_}}
> {{_File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch = super(MyMultiBranchPythonOperator, self).execute(context)_}}
> {{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 95, in execute return_value = self.execute_callable()_}}
> {{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 100, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs)_}}
> {{_File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining run('sudo echo -e "Host bitbucket.org\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config')_}}
> {{_File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True):_}}
> {{_File "/usr/lib/python3.5/subprocess.py", line 693, in run with Popen(*popenargs, **kwargs) as process:_}}
> {{_File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite) = self._get_handles(stdin, stdout, stderr)}}
> {{File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite = stdout.fileno()}}
> {{AttributeError: 'StreamLogWriter' object has no attribute 'fileno'}}
> {{[2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}
>
> {{Fixed locally by adding to the StreamLogWriter class:}}
> {{def fileno(self):}}
> {{ return 1}}
>
> {{(1 is stdout's file descriptor)}}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)