You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/09 17:42:26 UTC
[GitHub] [airflow] rafalh opened a new issue, #28810: Broken macro support in `DataProcJobBuilder` `task_id` parameter
rafalh opened a new issue, #28810:
URL: https://github.com/apache/airflow/issues/28810
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
8.5.0
### Apache Airflow version
2.3.4
### Operating System
Linux
### Deployment
Composer
### Deployment details
_No response_
### What happened
I am passing string containing macros to `DataProcJobBuilder` `task_id` parameter:
```
{{ (dag.dag_id + '-' + task.task_id.replace('_', '-'))[:90] }}
```
I use created job with `DataprocSubmitJobOperator`.
It worked fine in composer-2.0.27-airflow-2.2.5 (apache-airflow-providers-google==2022.9.6+composer). Now I tried upgrading to composer-2.1.2-airflow-2.3.4 (apache-airflow-providers-google==8.5.0) and it no longer works because after #23791 all "." are replaced by "_" before macro evaluation. It causes error:
```
'dag_dag_id' is undefined
```
### What you think should happen instead
Optimally `DataProcJobBuilder` should allow using macros as before.
PR that broke this usage was supposed to fix task groups (#23439). Perhaps that fix (string replacement) could be moved to a different place e.g. `DataprocJobBaseOperator` default value. This way API of DataProcJobBuilder would still allow macros.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Broken macro support in `DataProcJobBuilder` `task_id` parameter [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28810:
URL: https://github.com/apache/airflow/issues/28810#issuecomment-1895167072
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28810: Broken macro support in `DataProcJobBuilder` `task_id` parameter
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28810:
URL: https://github.com/apache/airflow/issues/28810#issuecomment-1376380257
The replacement should be done AFTER running JINJA. Anyone wants to provide a fix? (@eladkal :) ? )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org