You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/15 19:32:23 UTC
[GitHub] [airflow] RNHTTR opened a new pull request, #25102: Incorrect parallelism definition in config.yml
RNHTTR opened a new pull request, #25102:
URL: https://github.com/apache/airflow/pull/25102
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #25102: Incorrect parallelism definition in config.yml
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #25102:
URL: https://github.com/apache/airflow/pull/25102#issuecomment-1232301214
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on a diff in pull request #25102: Incorrect parallelism definition in config.yml
Posted by GitBox <gi...@apache.org>.
RNHTTR commented on code in PR #25102:
URL: https://github.com/apache/airflow/pull/25102#discussion_r922482060
##########
airflow/config_templates/config.yml:
##########
@@ -62,10 +62,9 @@
default: "SequentialExecutor"
- name: parallelism
description: |
- This defines the maximum number of task instances that can run concurrently per scheduler in
- Airflow, regardless of the worker count. Generally this value, multiplied by the number of
- schedulers in your cluster, is the maximum number of task instances with the running
- state in the metadata database.
+ This defines the maximum number of task instances that can run concurrently in
+ Airflow, regardless of the worker or scheduler count. This value represents the maximum number
+ of tasks that can be in the running or queued states in the metadata database.
Review Comment:
Apologies -- my Airflow deployment only had one scheduler, but I thought it had two.
However, this doesn't make a lot of sense to me. Why would the number of schedulers influence a parameter that's used to limit the amount of work that can be done in parallel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on a diff in pull request #25102: Incorrect parallelism definition in config.yml
Posted by GitBox <gi...@apache.org>.
RNHTTR commented on code in PR #25102:
URL: https://github.com/apache/airflow/pull/25102#discussion_r922482060
##########
airflow/config_templates/config.yml:
##########
@@ -62,10 +62,9 @@
default: "SequentialExecutor"
- name: parallelism
description: |
- This defines the maximum number of task instances that can run concurrently per scheduler in
- Airflow, regardless of the worker count. Generally this value, multiplied by the number of
- schedulers in your cluster, is the maximum number of task instances with the running
- state in the metadata database.
+ This defines the maximum number of task instances that can run concurrently in
+ Airflow, regardless of the worker or scheduler count. This value represents the maximum number
+ of tasks that can be in the running or queued states in the metadata database.
Review Comment:
Apologies -- the Airflow deployment I was testing on only had one scheduler, but I thought it had two. When I updated the scheduler count to 2, double the amount of tasks could be run in parallel.
However, this doesn't make a lot of sense to me. Why would the number of schedulers influence a parameter that's used to limit the amount of work that can be done in parallel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #25102: Incorrect parallelism definition in config.yml
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #25102:
URL: https://github.com/apache/airflow/pull/25102#discussion_r922466696
##########
airflow/config_templates/config.yml:
##########
@@ -62,10 +62,9 @@
default: "SequentialExecutor"
- name: parallelism
description: |
- This defines the maximum number of task instances that can run concurrently per scheduler in
- Airflow, regardless of the worker count. Generally this value, multiplied by the number of
- schedulers in your cluster, is the maximum number of task instances with the running
- state in the metadata database.
+ This defines the maximum number of task instances that can run concurrently in
+ Airflow, regardless of the worker or scheduler count. This value represents the maximum number
+ of tasks that can be in the running or queued states in the metadata database.
Review Comment:
this basicly reverts https://github.com/apache/airflow/pull/23768
Can you please share some information to backup the suggested description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a diff in pull request #25102: Incorrect parallelism definition in config.yml
Posted by GitBox <gi...@apache.org>.
mik-laj commented on code in PR #25102:
URL: https://github.com/apache/airflow/pull/25102#discussion_r922639514
##########
airflow/config_templates/config.yml:
##########
@@ -62,10 +62,9 @@
default: "SequentialExecutor"
- name: parallelism
description: |
- This defines the maximum number of task instances that can run concurrently per scheduler in
- Airflow, regardless of the worker count. Generally this value, multiplied by the number of
- schedulers in your cluster, is the maximum number of task instances with the running
- state in the metadata database.
+ This defines the maximum number of task instances that can run concurrently in
+ Airflow, regardless of the worker or scheduler count. This value represents the maximum number
+ of tasks that can be in the running or queued states in the metadata database.
Review Comment:
Because this value is not synchronized between schedulers due to performance reasons.. Each scheduler only monitors its own tasks, so everyone has their own quota.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #25102: Incorrect parallelism definition in config.yml
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #25102: Incorrect parallelism definition in config.yml
URL: https://github.com/apache/airflow/pull/25102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org