You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/09 17:37:08 UTC
[GitHub] [airflow] kaxil opened a new issue #20175: Add missing docs on using more than one `pool_slot`
kaxil opened a new issue #20175:
URL: https://github.com/apache/airflow/issues/20175
### Describe the issue with documentation
We added a feature that allow more than one pool slots to be used by a single task in https://github.com/apache/airflow/pull/7160
However, we currently have no docs on it.
### How to solve the problem
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990223979
> I get that, but what do pool slots give over creating a second pool with fewer slots?
Say we have:
`Pool(pool='db', slots=2)`
This pool protect a resource that can handle up to 2 units of computing power.
and we have:
```
with DAG(...) as dag:
BashOperator(
task_id='big_task',
bash_command='python script.py',
pool_slots=2,
pool='db'
)
BashOperator(
task_id='small_task1',
bash_command='python script2.py',
pool_slots=1,
pool='db'
)
with DAG(...) as dag2:
BashOperator(
task_id='small_task2',
bash_command='python script3.py',
pool_slots=1,
pool='db'
)
```
With this code the resource is protected according to requirements.
How would suggest to get the same functionality with 2 pools (without `pool_slots`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990295946
Ah. How cool I did not know Airflow can do that :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990223979
> I get that, but what do pool slots give over creating a second pool with fewer slots?
Say we have:
`Pool(pool='db', slots=2)`
This pool protect a resource that can handle up to 2 units of computing power.
and we have:
```
with DAG(...) as dag:
BashOperator(
task_id='task1',
bash_command='python script.py',
pool_slots=2,
pool='db'
)
BashOperator(
task_id='task2',
bash_command='python script2.py',
pool_slots=1,
pool='db'
)
with DAG(...) as dag2:
BashOperator(
task_id='task3',
bash_command='python script3.py',
pool_slots=1,
pool='db'
)
```
With this code the resource is protected according to requirements.
How would suggest to get the same functionality with 2 pools (without `pool_slots`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990223979
> I get that, but what do pool slots give over creating a second pool with fewer slots?
Say we have:
`Pool(pool='db', slots=2)`
This pool protect a resource that can handle up to 2 units of computing power.
and we have:
```
with DAG(...) as dag:
BashOperator(
task_id='task1',
bash_command='echo Big Task',
pool_slots=2,
pool='db'
)
BashOperator(
task_id='task2',
bash_command='echo Small Task',
pool_slots=1,
pool='db'
)
with DAG(...) as dag2:
BashOperator(
task_id='task3',
bash_command='echo Small Task',
pool_slots=1,
pool='db'
)
```
With this code the resource is protected according to requirements.
How would suggest to get the same functionality with 2 pools (without `pool_slots`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990192601
> Subscribing to the thread because I wonder what the use case is for pool slots.
Pools are used to limit concurrency with external services.
Not all tasks "weight" the same. Some tasks might require more resources from the external service - so for those heavy tasks you might want to occupy more slots of the pool to avoid scheduling more tasks. For example: allowing to run up to 2 small tasks or up to 1 big task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990192601
> Subscribing to the thread because I wonder what the use case is for pool slots.
Pools are used to limit concurrency with external services.
Not all tasks "weight" the same. Some tasks might require more resources from the external service - so for those heavy tasks you might want to occupy more slots of the pool to avoid scheduling more tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990192601
> Subscribing to the thread because I wonder what the use case is for pool slots.
Pools are used to limit concurrency with external services.
Not all tasks "weight" the same. Some tasks might require more resources from the external service - so for those heavy tasks you might want to occupy more slots of the pool to avoid scheduling more tasks. For example: allowing run 2 small tasks or 1 big task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #20175:
URL: https://github.com/apache/airflow/issues/20175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-991928940
Feel free!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] PApostol commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
PApostol commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-991924813
I can take this one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990209572
I get that, but what do pool slots give over creating a second pool with fewer slots?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-990175731
Subscribing to the thread because I wonder what the use case is for pool slots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #20175: Add missing docs on using more than one `pool_slot`
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #20175:
URL: https://github.com/apache/airflow/issues/20175#issuecomment-992448222
Fixed by https://github.com/apache/airflow/pull/20257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org