You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/05 10:36:15 UTC
[GitHub] [airflow] codenamestif opened a new issue #16808: Wrong template field types for AWS operators
codenamestif opened a new issue #16808:
URL: https://github.com/apache/airflow/issues/16808
**Apache Airflow version**: 2.1.1
**Environment**:
- **Cloud provider or hardware configuration**: AWS
**What happened**:
After migrating from Airflow 2.0.1 to 2.1.1, the Rendered Template tab stopped working for dictionaries. I checked it for Sagemaker and ECS operators, the code is not shown.
<img width="400" src="https://user-images.githubusercontent.com/2885779/124456386-ead53a00-dd8a-11eb-9b30-f1a627c8f8f2.png">
However for some operators where i have customized list of `template_fields`, it partly works (only for extra fields).
<img width="400" alt="Screenshot 2021-07-05 at 12 18 37" src="https://user-images.githubusercontent.com/2885779/124457148-c9288280-dd8b-11eb-9c2c-082bbafaa99c.png">
So then i checked a release notes and it looks like it's related to #15130 and for some AWS operators the type of the fields should be not `py`, but `json` since all those fields mostly simply propagated to boto3 as json.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jordanjeremy edited a comment on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
jordanjeremy edited a comment on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-876615292
As mentioned previously, this is not specific to AWS operators. This issue also happens with @task decorated (TaskFlow API) operators, where the op_kwargs is a dict and is setup to be rendered as py.
Possible solution is to update the ["get_python_source" in utils/code_utils.py](https://github.com/apache/airflow/blob/main/airflow/utils/code_utils.py#L27) to handle list and dict. Possibly something like:
```
if isinstance(x, list):
return [str(v) for v in x]
if isinstance(x, dict):
return {k: str(v) for k, v in x.items()}
```
The converting values to strings seems necessary in order to avoid errors when this is passed to the pygments lexer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-874008341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-874029548
@uranusjr i can make pr but only for the operators i use
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-877821005
Closed by https://github.com/apache/airflow/pull/16820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #16808:
URL: https://github.com/apache/airflow/issues/16808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-874017265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-874017265
Hi! Would you be interested in submitting a PR to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jordanjeremy commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
jordanjeremy commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-876615292
As mentioned previously, this is not specific to AWS operators. This issue also happens with @task decorated (TaskFlow API) operators, where the op_kwargs is a dict and is setup to be rendered as py.
Possible solution is to update the "get_python_source" in utils/code_utils.py to handle list and dict. Possibly something like:
```
if isinstance(x, list):
return [str(v) for v in x]
if isinstance(x, dict):
return {k: str(v) for k, v in x.items()}
```
The converting values to strings seems necessary in order to avoid errors when this is passed to the pygments lexer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-874061876
That’s perfectly fine! If some other operators also have this issue, people who use them can submit fixes for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #16808: Wrong template field types for AWS operators
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #16808:
URL: https://github.com/apache/airflow/issues/16808#issuecomment-874008341
It might be not just for AWS operators, but these are the ones i use and i could test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org