You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/28 01:46:16 UTC
[GitHub] [airflow] kaxil opened a new pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
kaxil opened a new pull request #15557:
URL: https://github.com/apache/airflow/pull/15557
Currently, the `serve_logs` endpoint only exists on Celery workers. This
means if someone launches Airflow with the `LocalExecutor` and wants to
grab the logs from the scheduler, there is no way to move that to the
webserver if it is on a different pod/machine.
This commit makes the scheduler automatically serves logs when using
`LocalExecutor` or `SequentialExecutor`. However, it means for
Airflow <= 2.0.2, the Helm Chart won't serve logs.
closes https://github.com/apache/airflow/pull/15070
closes https://github.com/apache/airflow/issues/13331
closes https://github.com/apache/airflow/issues/15071
closes https://github.com/apache/airflow/issues/14222
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #15557:
URL: https://github.com/apache/airflow/pull/15557#discussion_r622591953
##########
File path: airflow/cli/commands/scheduler_command.py
##########
@@ -30,6 +32,8 @@
@cli_utils.action_logging
def scheduler(args):
"""Starts Airflow Scheduler"""
+ skip_serve_logs = args.skip_serve_logs
Review comment:
This is only ON by default for `LocalExecutor` and `SequentialExecutor` though. And kept this behaviour to replicate what we have on it our Worker (`airflow celery worker`) - https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#worker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #15557:
URL: https://github.com/apache/airflow/pull/15557#issuecomment-828078108
Alternative approach: https://github.com/apache/airflow/pull/15556 where Helm Chart will work pre and post 2.0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #15557:
URL: https://github.com/apache/airflow/pull/15557#discussion_r622591953
##########
File path: airflow/cli/commands/scheduler_command.py
##########
@@ -30,6 +32,8 @@
@cli_utils.action_logging
def scheduler(args):
"""Starts Airflow Scheduler"""
+ skip_serve_logs = args.skip_serve_logs
Review comment:
This is only ON by default for `LocalExecutor` and `SequentialExecutor` though. And kept this behaviour to replicate what we have on it our Worker (`airflow celery worker`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #15557:
URL: https://github.com/apache/airflow/pull/15557#discussion_r622599685
##########
File path: airflow/cli/commands/scheduler_command.py
##########
@@ -30,6 +32,8 @@
@cli_utils.action_logging
def scheduler(args):
"""Starts Airflow Scheduler"""
+ skip_serve_logs = args.skip_serve_logs
Review comment:
Ah, consistency with worker makes sense. I don't have a strong opinion here, just when looking at scheduler in isolation the default seems backwards to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #15557:
URL: https://github.com/apache/airflow/pull/15557#discussion_r622532036
##########
File path: airflow/cli/commands/scheduler_command.py
##########
@@ -30,6 +32,8 @@
@cli_utils.action_logging
def scheduler(args):
"""Starts Airflow Scheduler"""
+ skip_serve_logs = args.skip_serve_logs
Review comment:
I wonder if this should be the opposite - opt in to log serving. There is probably more usage that wouldn't want this on by default than the other way around (e.g. running locally).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #15557: Serve logs with Scheduler when using Local or Sequential Executor
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #15557:
URL: https://github.com/apache/airflow/pull/15557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org