You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "royassis (via GitHub)" <gi...@apache.org> on 2025/11/05 08:33:37 UTC

[D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

GitHub user royassis created a discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

When trying to create a PythonVirtualenvOperator with a dynamic executor_config i am getting 

```shell
TypeError: unmappable or already specified argument: executor_config
```

is this a "feature" or a bug?

GitHub link: https://github.com/apache/airflow/discussions/57850

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a comment on the discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

@potiuk here's my issue.

I am using `PythonVirtualenvOperator` with `AwsBatchExecutor` which runs the task inside an AWS Batch job.
I would like to dynamiclly tag the job with the `dag_id` and `run_id` also set the jobName dynamiclly by runtime params. 

But it seems that executor_config don't support it. 

The real issue is that I can't connect the airflow task with the AWS Batch job run.

GitHub link: https://github.com/apache/airflow/discussions/57850#discussioncomment-14880902

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a comment on the discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

@potiuk I'll tell you my issue.

I am using `PythonVirtualenvOperator` with `AwsBatchExecutor` which runs the task inside an AWS Batch job.
I would like to dynamiclly tag the job with the `dag_id` and `run_id` also set the jobName dynamiclly by runtime params. 

But it seems that executor_config don't support it. 

The real issue is that I can't connect the airflow task with the AWS Batch job run.

GitHub link: https://github.com/apache/airflow/discussions/57850#discussioncomment-14880902

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

When trying to create a PythonVirtualenvOperator with a dynamic `executor_config` i am getting 

```shell
TypeError: unmappable or already specified argument: executor_config
```

is this a "feature" or a bug?

GitHub link: https://github.com/apache/airflow/discussions/57850

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a comment on the discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

@potiuk here's my issue.

I am using `PythonVirtualenvOperator` with `AwsBatchExecutor` which runs the task inside an AWS Batch job.
I would like to dynamiclly tag the job with the `dag_id` and `run_id` also set the jobName dynamiclly by runtime params. 

But it seems that executor_config don't support it. 

The real issue is that I can't connect the airflow task with the AWS Batch job run.

EDIT:
Also the `executor_config` contains this configuration:

```python
"containerOverrides": {
                "resourceRequirements": [
                    {"type": "VCPU", "value": "2"},
                    {"type": "MEMORY", "value": "4192"},
                ]
            }
```

For setting the mem and cpus for the Aws Batch jobs, and this can't be set dynamiclly. 

GitHub link: https://github.com/apache/airflow/discussions/57850#discussioncomment-14880902

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

When trying to create a PythonVirtualenvOperator with a dynamic `executor_config` i am getting 

```shell
TypeError: unmappable or already specified argument: executor_config
```

```python
task_a = PythonVirtualenvOperator.partial(
    task_id="task_a",
    ...
).expand(
    executor_config=[{}])
)
```

is this a "feature" or a bug?

GitHub link: https://github.com/apache/airflow/discussions/57850

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis added a comment to the discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

@potiuk I'll tell you my issue.

I am using `PythonVirtualenvOperator` with `AwsBatchExecutor` which runs the task inside an AWS Batch job.
I would like to dynamiclly tag the job with the `dag_id` and `run_id` also set the jobName dynamiclly by runtime params. 

But it seems that executor_config don't support it. 

They real issue is that I can't connect the airflow task with the AWS Batch job run.

GitHub link: https://github.com/apache/airflow/discussions/57850#discussioncomment-14880902

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

When trying to create a PythonVirtualenvOperator with a dynamic `executor_config` i am getting 

```shell
TypeError: unmappable or already specified argument: executor_config
```

```python
task_a = PythonVirtualenvOperator.partial(
    task_id="task_a",
    ...
).expand(
    executor_config=generate_executor_config.expand(task_name=[{}])
)
```

is this a "feature" or a bug?

GitHub link: https://github.com/apache/airflow/discussions/57850

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis edited a discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

When trying to create a PythonVirtualenvOperator with a dynamic `executor_config` i am getting 

```shell
TypeError: unmappable or already specified argument: executor_config
```

```python
task_a = PythonVirtualenvOperator.partial(
    task_id="task_a",
    ...
).expand(
    executor_config=[{}]
)
```

is this a "feature" or a bug?

GitHub link: https://github.com/apache/airflow/discussions/57850

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

GitHub user potiuk added a comment to the discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

it's a feature. You can define single executor config for all mapped tasks together, you cannot individually specify different configuration for different mapped tasks. This is how executors work internally - they map tasks *after* executor config is applied.

GitHub link: https://github.com/apache/airflow/discussions/57850#discussioncomment-14879499

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] When dynamic mapping of PythonVirtualenvOperator executor_config [airflow]

Posted by "royassis (via GitHub)" <gi...@apache.org>.

GitHub user royassis closed a discussion: When dynamic mapping of PythonVirtualenvOperator executor_config

When trying to create a PythonVirtualenvOperator with a dynamic `executor_config` i am getting 

```shell
TypeError: unmappable or already specified argument: executor_config
```

```python
task_a = PythonVirtualenvOperator.partial(
    task_id="task_a",
    ...
).expand(
    executor_config=[{}]
)
```

is this a "feature" or a bug?

GitHub link: https://github.com/apache/airflow/discussions/57850

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org