[GitHub] [airflow] Deelane opened a new issue, #28526: Not sure if bug? BashOperator within dynamic mapped airflow tasks not using specified pool

[GitBox <gi...@apache.org>]

Deelane opened a new issue, #28526:
URL: https://github.com/apache/airflow/issues/28526

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   Airflow version: 2.3.3
   
   We are starting to transition to task decorators from Operators in our dags and I have ran into a use case that I could not find any resources for, and I am unsure if it is a bug.
   
   I am attempting to invoke a task multiple times via dynamic mapping like this:
   
   ```
   @task_group
   def some_task_group():
     if condition: 
             some_mapped_task.partial(
                 partial_arg=some_arg
             ).expand(
                 expand_list_arg=some_list)
             ) 
   ```
   
   
   Here, `some_mapped_task` is the mapped task. Within `some_mapped_task` exists a function `execute_bash_function` that invokes a regular `BashOperator` and passes a task pool to run on:
   
   ```
   @task
   def some_mapped_task():
     task_pool=get_task_pool()
     execute_bash_function(task_pool=task_pool)
   ```
   
   ```
   def execute_bash_function(task_pool: str):
       cmd = f'some_bash_command`
       context = get_current_context()
       BashOperator(
           task_id=f'some_task_id',
           bash_command=cmd,
           pool=task_pool
       ).execute(context)
   ```
   
   So we have a mapped task invoked multiple times via expand, and the mapped task itself attempts to call a function that executes a `BashOperator`. However, the `BashOperator` is not using the `task_pool` passed in and is instead using the default task pool that the parent task `some_mapped_task` uses. 
   
   I am not entirely certain if this is due to my lack of understanding or a bug with airflow. When assigning the `BashOperator` to a variable `operator` and printing `operator.pool`, the correct pool is printed. However, the default pool is used.
   
   ### What you think should happen instead
   
   If this is indeed a bug, then I believe the BashOperator should use the task pool passed in, and not the task pool of the parent task.
   
   ### How to reproduce
   
   - Create a `@task_group` 
   - Create a `@task`
   - Create a function that executes a `BashOperator`
    - Execute the `BashOperator` function within the `@task` 
    - Execute the `@task` within the task group, using `task_name.partial().expand()`
   
   ### Operating System
   
   MacOs Ventura 13.0
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow==2.3.3
   apache-airflow-client==2.2.0
   apache-airflow-providers-google==8.1.0
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #28526: Not sure if bug? BashOperator within dynamic mapped airflow tasks not using specified pool

[GitBox <gi...@apache.org>]

potiuk closed issue #28526: Not sure if bug? BashOperator within dynamic mapped airflow tasks not using specified pool
URL: https://github.com/apache/airflow/issues/28526


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #28526: Not sure if bug? BashOperator within dynamic mapped airflow tasks not using specified pool

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #28526:
URL: https://github.com/apache/airflow/issues/28526#issuecomment-1362058739

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #28526: Not sure if bug? BashOperator within dynamic mapped airflow tasks not using specified pool

[GitBox <gi...@apache.org>]

potiuk commented on issue #28526:
URL: https://github.com/apache/airflow/issues/28526#issuecomment-1362065465

   This is very wrong use of operator. You are not supposed to create BashOperator from inside a task - task is already an operator and running BashOperator inside is just creating an object and running execute on it.
   
   The BashOperator must be instantiated in the DAG parsing time to become a task - not during execution of the task method.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org