You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/22 21:14:41 UTC

[GitHub] [airflow] hsnprsd opened a new issue #16001: Can't view running tasks logs using KubernetesExecutor.

hsnprsd opened a new issue #16001:
URL: https://github.com/apache/airflow/issues/16001


   **Description**
   
   I want to be able to see logs for running tasks in airflow web UI when I'm using "KubernetesExecutor".
   Airflow tries to read logs of worker pod which does not contain my task logs because "airflow tasks run" command redirects all stdout/stderr to log file.
   
   Here is what I see when I try to read logs of running tasks:
   ```
   *** Falling back to local log
   *** Trying to get logs (last 100 lines) from worker pod datainfraexporterhourlydagwaitforpublishedpostdump.a825336790e24b40ad43b2411ef617c5 ***
   
   BACKEND=postgres
   DB_HOST=divar-infra-db-cluster-master-service.skubel-76f219d5-eb73-4f23-96b3-a8cf4ab4f00a
   DB_PORT=5432
   
   [2021-05-22 19:00:30,157] {dagbag.py:451} INFO - Filling up the DagBag from /git/airflow/dags/data_infra/exporter/hourly.py
   ```
   
   **Use case / motivation**
   
   It's not convenient to wait for tasks to finish in order to read their logs.
   
   **Are you willing to submit a PR?**
   
   I can help.
   
   **Related Issues**
   
   I don't think so.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846535861


   @andormarkus I said. It does not work without remote logging either.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] deas commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
deas commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-854702500


   I guess we don't want to break expected default behavior.
   
   However, I think we should allow container users to switch to "all logs stdout" when using the `apache/airflow` images leveraging `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jherrmannNetfonds commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
jherrmannNetfonds commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-854674139


   It would be great if there was an option that the task log to stdout (at least for kubernetesExecutor). Writing to stdout on kubernetesExecutor probably would make remote logging obsolete in most cases, since the cloud logging tools on GCP/AWS/Azure collect the logs for kubernetes anyway.
   Does anybody now if somebody is working on this? I would help, but not sure where to start here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846534958


   Here it is:
   AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://airflow-logs
   AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: airflow-logs
   AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
   
   But it does not matter. I tested it without remote logging and it does not work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd edited a comment on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd edited a comment on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846535185


   I came up with a solution today.
   I changed the default logging config class like this:
   
   ```
   import sys
   from copy import deepcopy
   
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   
   LOGGING_CONFIG["handlers"]["custom_console"] = {
       "class": "logging.StreamHandler",
       "formatter": "airflow",
       "stream": sys.__stdout__,
   }
   LOGGING_CONFIG["loggers"]["airflow.task"]["handlers"].append("custom_console")
   ```
   
   I moved this file to `$AIRFLOW_HOME/config/logging_config.py` and changed `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS` to `logging_config.LOGGING_CONFIG`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] andormarkus commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846534163


   Hi @hsnprsd 
   Please can you share your logs configuration?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd edited a comment on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd edited a comment on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846534958


   Here it is:
   ```
   AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://airflow-logs
   AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: airflow-logs
   AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
   ```
   
   But it does not matter. I tested it without remote logging locally and it does not work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] whitleykeith commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
whitleykeith commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-904798933


   I've had this issue for a while now, but I was able to resolve it by adding this to the top of my `dag.py` file.
   
   
   ```python
   # Set Task Logger to INFO for better task logs
   log = logging.getLogger("airflow.task")
   handler = logging.StreamHandler(sys.stdout)
   handler.setLevel(logging.INFO)
   log.addHandler(handler)
   ```
   
   It does pretty much what @hsnprsd had in his class, but doesn't require overriding the default logging config. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] andormarkus commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846535718


   If you are using remote logging then "Note that logs are only sent to remote storage once a task is complete (including failure); In other words, remote logs for running tasks are unavailable (but local logs are available)." [source](https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846535185


   I came up with a solution today.
   I changed the default logging config class like this:
   
   ```
   import sys
   from copy import deepcopy
   
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   
   LOGGING_CONFIG["handlers"]["custom_console"] = {
       "class": "logging.StreamHandler",
       "formatter": "airflow",
       "stream": sys.__stdout__,
   }
   LOGGING_CONFIG["loggers"]["airflow.task"]["handlers"].append("custom_console")
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] deas edited a comment on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
deas edited a comment on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-854702500


   I guess we don't want to break expected default behavior.
   
   However, I think we should allow container users to switch to "all logs stdout" when using the `apache/airflow` images leveraging `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS` by providing code close to what @hsnprsd mentioned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd edited a comment on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd edited a comment on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846535185


   I came up with a solution today.
   I changed the default logging config class like this:
   
   ```
   import sys
   from copy import deepcopy
   
   from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
   
   LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
   
   LOGGING_CONFIG["handlers"]["custom_console"] = {
       "class": "logging.StreamHandler",
       "formatter": "airflow",
       "stream": sys.__stdout__,
   }
   LOGGING_CONFIG["loggers"]["airflow.task"]["handlers"].append("custom_console")
   ```
   
   I moved this file to $AIRFLOW_HOME/config/logging_config.py and changed `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS` to `logging_config.LOGGING_CONFIG`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] asimerel commented on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
asimerel commented on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846539494


   Hi @andormarkus ,
   I have the same problem, it is not about remote logging. Remote logs are available after the task completed as expected.
   The logs are not available while the task is running with KubernetesExecutor from UI because UI is looking for pod logs of the kubernetes pod that running "airflow task run" command. But that pod doesn't expose task logs to console. So UI only see Filling up the DagBag log like @hsnprsd attached on the first messeage until the task is completed and logs retrieved from remote.
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hsnprsd edited a comment on issue #16001: Can't view running tasks logs using KubernetesExecutor.

Posted by GitBox <gi...@apache.org>.
hsnprsd edited a comment on issue #16001:
URL: https://github.com/apache/airflow/issues/16001#issuecomment-846534958


   Here it is:
   ```
   AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://airflow-logs
   AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: airflow-logs
   AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
   ```
   
   But it does not matter. I tested it without remote logging and it does not work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org