You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/09 21:50:54 UTC
[GitHub] [airflow] MaxTaggart opened a new pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
MaxTaggart opened a new pull request #14160:
URL: https://github.com/apache/airflow/pull/14160
related: #13805
This is to address issue #13805 where when using the Kubernetes executor backfill jobs or jobs triggered from the UI will produce this error:
```
[2021-01-27 06:35:50,376] {kubernetes_executor.py:491} INFO - Start Kubernetes executor
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 116, in dag_backfill
run_backwards=args.run_backwards,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dag.py", line 1701, in run
job.run()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/backfill_job.py", line 788, in _execute
executor.start()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
```
The fix is to simply add a hardcoded executor `job_id` in `airflow/jobs/backfill_job.py` and `airflow/www/views.py`.
The changes still need tests, but I figured that this PR would be a good place to discuss where those tests should live (I'm not familiar with the testing landscape yet).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-777193912
[The Workflow run](https://github.com/apache/airflow/actions/runs/556768814) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
dimberman commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-777046609
@MaxTaggart it looks like the test is failing due to the test on views.
I think the test on backfill should be sufficient. I don't think there's an easy way to run the views test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-777194102
[The Workflow run](https://github.com/apache/airflow/actions/runs/556719569) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-776909599
[The Workflow run](https://github.com/apache/airflow/actions/runs/555424666) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-776267271
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-781308417
```
2021-02-16T12:50:30.8256733Z tests/www/api/experimental/test_dag_runs_endpoint.py ssssss [ 99%]
2021-02-16T13:27:41.4105606Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
```
My that's a _large_ gap between those to lines.
@MaxTaggart Could you rebase and try again? We had a few problems with CI over last few days that should be better now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-777226926
[The Workflow run](https://github.com/apache/airflow/actions/runs/556970310) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-776975038
[The Workflow run](https://github.com/apache/airflow/actions/runs/555749507) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #14160:
URL: https://github.com/apache/airflow/pull/14160
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] MaxTaggart commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
MaxTaggart commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-777064167
@dimberman Are you saying to drop the views test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-776867172
[The Workflow run](https://github.com/apache/airflow/actions/runs/555319657) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-786139422
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14160: Unable to trigger backfill or manual jobs with Kubernetes executor.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14160:
URL: https://github.com/apache/airflow/pull/14160#issuecomment-776316088
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org