You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/10/25 13:45:21 UTC
[GitHub] [airflow] mik-laj commented on issue #6389: [AIRFLOW-5663]: Switch
to real-time logging in PythonVirtualenvOperator.
mik-laj commented on issue #6389: [AIRFLOW-5663]: Switch to real-time logging in PythonVirtualenvOperator.
URL: https://github.com/apache/airflow/pull/6389#issuecomment-546360165
Hello,
I used the example below.
```
From 860d5addc8ab6ffec9327a4a861bf7547ea9d5cc Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kamil=20Bregu=C5=82a?= <ka...@polidea.com>
Date: Fri, 25 Oct 2019 15:43:42 +0200
Subject: Add example for PythonVirtualenvOperator
---
.../example_dags/example_python_operator.py | 26 ++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)
diff --git a/airflow/example_dags/example_python_operator.py b/airflow/example_dags/example_python_operator.py
index 86403ceb2..68e617731 100644
--- a/airflow/example_dags/example_python_operator.py
+++ b/airflow/example_dags/example_python_operator.py
@@ -24,7 +24,7 @@ from pprint import pprint
import airflow
from airflow.models import DAG
-from airflow.operators.python_operator import PythonOperator
+from airflow.operators.python_operator import PythonOperator, PythonVirtualenvOperator
args = {
'owner': 'Airflow',
@@ -71,3 +71,27 @@ for i in range(5):
run_this >> task
# [END howto_operator_python_kwargs]
+
+
+def callable_virtualenv():
+ from colorama import Fore, Back, Style
+ from time import sleep
+ print(Fore.RED + 'some red text')
+ print(Back.GREEN + 'and with a green background')
+ print(Style.DIM + 'and in dim text')
+ print(Style.RESET_ALL)
+ for _ in range(10):
+ print(Style.DIM + 'Please wait...')
+ sleep(10)
+ print('Finished')
+
+
+virtualenv_task = PythonVirtualenvOperator(
+ task_id="virtualenv_python",
+ python_callable=callable_virtualenv,
+ requirements=[
+ "colorama==0.4.0"
+ ],
+ system_site_packages=False,
+ dag=dag,
+)
--
2.20.1
```
Unfortunately, the messages do not show in real time. Did I miss something?
```
root@1ddb884f5c23:/opt/airflow# airflow tasks test example_python_operator virtualenv_python 2019-01-01
/opt/airflow/airflow/models/dagbag.py:21: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
[2019-10-25 13:38:59,355] {settings.py:175} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=57962
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
[2019-10-25 13:38:59,893] {__init__.py:53} INFO - Using executor LocalExecutor
[2019-10-25 13:38:59,893] {dagbag.py:88} INFO - Filling up the DagBag from /opt/airflow/airflow/example_dags
[2019-10-25 13:39:00,399] {taskinstance.py:632} INFO - Dependencies all met for <TaskInstance: example_python_operator.virtualenv_python 2019-01-01T00:00:00+00:00 [None]>
[2019-10-25 13:39:00,410] {taskinstance.py:632} INFO - Dependencies all met for <TaskInstance: example_python_operator.virtualenv_python 2019-01-01T00:00:00+00:00 [None]>
[2019-10-25 13:39:00,410] {taskinstance.py:843} INFO -
--------------------------------------------------------------------------------
[2019-10-25 13:39:00,410] {taskinstance.py:844} INFO - Starting attempt 1 of 1
[2019-10-25 13:39:00,411] {taskinstance.py:845} INFO -
--------------------------------------------------------------------------------
[2019-10-25 13:39:00,413] {taskinstance.py:864} INFO - Executing <Task(PythonVirtualenvOperator): virtualenv_python> on 2019-01-01T00:00:00+00:00
[2019-10-25 13:39:00,428] {python_operator.py:135} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=example_python_operator
AIRFLOW_CTX_TASK_ID=virtualenv_python
AIRFLOW_CTX_EXECUTION_DATE=2019-01-01T00:00:00+00:00
[2019-10-25 13:39:00,429] {python_operator.py:334} INFO - Executing cmd
['virtualenv', '/tmp/venvnw9fmnxz']
[2019-10-25 13:39:00,439] {python_operator.py:340} INFO - Got output
[2019-10-25 13:39:01,674] {python_operator.py:343} INFO - b"Using base prefix '/usr/local'\n"
[2019-10-25 13:39:01,676] {python_operator.py:343} INFO - b'New python executable in /tmp/venvnw9fmnxz/bin/python\n'
[2019-10-25 13:39:01,676] {python_operator.py:343} INFO - b'Installing setuptools, pip, wheel...\n'
[2019-10-25 13:39:05,634] {python_operator.py:343} INFO - b'done.\n'
[2019-10-25 13:39:05,648] {python_operator.py:334} INFO - Executing cmd
['/tmp/venvnw9fmnxz/bin/pip', 'install', 'colorama==0.4.0']
[2019-10-25 13:39:05,656] {python_operator.py:340} INFO - Got output
[2019-10-25 13:39:06,315] {python_operator.py:343} INFO - b'Collecting colorama==0.4.0\n'
[2019-10-25 13:39:06,474] {python_operator.py:343} INFO - b' Downloading https://files.pythonhosted.org/packages/0a/93/6e8289231675d561d476d656c2ee3a868c1cca207e16c118d4503b25e2bf/colorama-0.4.0-py2.py3-none-any.whl\n'
[2019-10-25 13:39:06,523] {python_operator.py:343} INFO - b'Installing collected packages: colorama\n'
[2019-10-25 13:39:06,542] {python_operator.py:343} INFO - b'Successfully installed colorama-0.4.0\n'
[2019-10-25 13:39:06,824] {python_operator.py:334} INFO - Executing cmd
['/tmp/venvnw9fmnxz/bin/python', '/tmp/venvnw9fmnxz/script.py', '/tmp/venvnw9fmnxz/script.in', '/tmp/venvnw9fmnxz/script.out', '/tmp/venvnw9fmnxz/string_args.txt']
[2019-10-25 13:39:06,832] {python_operator.py:340} INFO - Got output
[2019-10-25 13:40:46,901] {python_operator.py:343} INFO - b'\x1b[31msome red text\n'
[2019-10-25 13:40:46,902] {python_operator.py:343} INFO - b'\x1b[42mand with a green background\n'
[2019-10-25 13:40:46,904] {python_operator.py:343} INFO - b'\x1b[2mand in dim text\n'
[2019-10-25 13:40:46,904] {python_operator.py:343} INFO - b'\x1b[0m\n'
[2019-10-25 13:40:46,905] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,905] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,906] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,907] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,907] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,908] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,908] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,908] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,909] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,909] {python_operator.py:343} INFO - b'\x1b[2mPlease wait...\n'
[2019-10-25 13:40:46,910] {python_operator.py:343} INFO - b'Finished\n'
[2019-10-25 13:40:46,959] {python_operator.py:144} INFO - Done. Returned value was: None
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services