[GitHub] [airflow] xinbinhuang edited a comment on issue #14813: Forked processes aren't logged by tasks using the Python Operator

[GitBox <gi...@apache.org>]

xinbinhuang edited a comment on issue #14813:
URL: https://github.com/apache/airflow/issues/14813#issuecomment-820876982


   Hi @rbankston, I don't think this is an airflow bug, but just how the `subprocess` and the `logging` module work. In a word, you need to propagate back the `stdout` from your subprocess back in order to log in airflow. There are two options:
   
   1. propagate the `stdout` yourself, i.e.:
   ```python
       ...
       p = subprocess.run(shlex.split(cmd), capture_output=True)
       print(p.stdout)
       ret = p.returncode
       ...
   ```
   In this case, the `p.stdout` is a simple string and you may need to parse it yourself.
   
   2. Use the `SubprocessHook` (recommended). This will automatically capture the stdout and log them in airflow for you.
   
   ```python
   sub_result:  SubprocessResult = SubprocessHook().run_command(shlex.split(cmd))
   
   print(sub_result.exit_code)
   
   # this is only necessary if you want to do extra stuff with the log other than logging in airflow
   print(sub_result.output)
   ```
   
   cc: @dstandish 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org