You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/30 07:05:35 UTC

[GitHub] [airflow] surapuramakhil opened a new issue #17332: different parallelism for different types of task

surapuramakhil opened a new issue #17332:
URL: https://github.com/apache/airflow/issues/17332


   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   **Description**
   
   <!-- A short description of your feature -->
   
   There could be certain task which requires huge amount of resources which can't be run with default parallelism and many other smaller tasks which are can run at parallelism of 32.
   
   At present there is a parallelism config
   
   The amount of parallelism as a setting to the executor. This defines the max number of task instances that should run simultaneously on this airflow installation 
   parallelism = 32
   
   A way where we can tag tasks and different level of parallelism for different tasks at entire airflow level.
   
   Like having smaller task to run at default parallelism [32] but heavy task at much lower parallelism [1-4]
   
   **Use case / motivation**
   
   <!-- What do you want to happen?
   
   Rather than telling us how you might implement this solution, try to take a
   step back and describe what you are trying to achieve.
   
   -->
   
   **Are you willing to submit a PR?**
   
   <!--- We accept contributions! -->
   
   **Related Issues**
   
   <!-- Is there currently another issue associated with this? -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #17332: different parallelism for different types of task

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17332:
URL: https://github.com/apache/airflow/issues/17332


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #17332: different parallelism for different types of task

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17332:
URL: https://github.com/apache/airflow/issues/17332#issuecomment-889792033


   This is currently possible with "pools" feature. https://airflow.apache.org/docs/apache-airflow/stable/concepts/pools.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org