You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/11 17:05:14 UTC
[GitHub] [airflow] ferruzzi opened a new pull request #21523: Update EKS sample DAGs and docs
ferruzzi opened a new pull request #21523:
URL: https://github.com/apache/airflow/pull/21523
Part of a project to standardize AWS provider package sample DAGs and documentation.
- Adds entries in the docs for the three EKS State Sensors
- All EKS sample DAGs are reformatted to be more consistent and to make the embedded snippets in the docs more approachable
- There is an example DAG that shows off how to use Jinja templates but it didn't actually use them for much, so I made that a better demo.
related: https://github.com/apache/airflow/pull/21475
repost of https://github.com/apache/airflow/pull/21508 after fixing the git bork.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r805323142
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
@ferruzzi you get the error because this line doesn't do what you expect.
You expect it `"{{ dag_run.conf['resources_vpc_config'] }}"` to be the Json define in the beginning of the file:
```
{
"subnetIds": ["subnet-12345ab", "subnet-67890cd"],
"endpointPublicAccess": true,
"endpointPrivateAccess": false
}
```
But this is not what is happening. The Jinja engine runs only in templated fields of operators. It does not run in top level python code. So when you execute `json.loads` you actually execute it on the string that you wrote.
If you will run in your python console :
`json.loads("{{ dag_run.conf['resources_vpc_config'] }}")`
You will see it will give you :
`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r805323142
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
@ferruzzi you get the error because this line doesn't do what you expect.
You expect it `"{{ dag_run.conf['resources_vpc_config'] }}"` to be the Json define in the beginning of the file:
```
{
"subnetIds": ["subnet-12345ab", "subnet-67890cd"],
"endpointPublicAccess": true,
"endpointPrivateAccess": false
}
```
But this is not what is happening. The Jinja engine runs only in templated fields of operators. It does not run in top level python code. So when you execute `json.loads` you actually execute it on the string that you wrote.
If you will run in your python console :
`json.loads("{{ dag_run.conf['resources_vpc_config'] }}")`
You will see it will give you :
`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)`
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
@ferruzzi you get the error because this line doesn't do what you expect.
You expect `"{{ dag_run.conf['resources_vpc_config'] }}"` to be the Json define in the beginning of the file:
```
{
"subnetIds": ["subnet-12345ab", "subnet-67890cd"],
"endpointPublicAccess": true,
"endpointPrivateAccess": false
}
```
But this is not what is happening. The Jinja engine runs only in templated fields of operators. It does not run in top level python code. So when you execute `json.loads` you actually execute it on the string that you wrote.
If you will run in your python console :
`json.loads("{{ dag_run.conf['resources_vpc_config'] }}")`
You will see it will give you :
`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)`
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
@ferruzzi you get the error because this line doesn't do what you expect.
You expect `"{{ dag_run.conf['resources_vpc_config'] }}"` to be the Json define in the beginning of the file:
```
{
"subnetIds": ["subnet-12345ab", "subnet-67890cd"],
"endpointPublicAccess": true,
"endpointPrivateAccess": false
}
```
But this is not what is happening. The Jinja engine runs only in templated fields of operators. It does not run in top level python code. So when you execute `json.loads` you actually execute it on the string that you wrote since it wasn't replaced with the Json defined earlier.
If you will run in your python console :
`json.loads("{{ dag_run.conf['resources_vpc_config'] }}")`
You will see it will give you :
`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
eladkal merged pull request #21523:
URL: https://github.com/apache/airflow/pull/21523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r805616731
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
To render as native Python object you will need `render_template_as_native_obj=True`
docs https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html#rendering-fields-as-native-python-objects.
The templating is done as part of creating a task instance:
https://github.com/apache/airflow/blob/fded2ca0b9c995737b401896b89e5c9fd7f24c91/airflow/models/taskinstance.py#L1419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r805539267
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
Thanks. Yeah, that is what I was gathering but I haven't come across an answer yet. Jinja is new to me so this may be really obvious. Does/can jinja magically know it should be a dict, or do I change the operator to accept type Union[dict, str], and if it's a string then handle the conversion inside the operator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r807172264
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
Should be fixed in https://github.com/apache/airflow/pull/21523/commits/bc1bca40db832675533ec2a65d2a28dca9f5cffe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#issuecomment-1036451314
having a jinja issue I need to figure out. If anyone knows the solution offhand, that would be appreciated, otherwise I'm reading up on it and will get it sooner or later.
```
File "/opt/airflow/airflow/providers/amazon/aws/example_dags/example_eks_templated.py", line 58, in <module>
VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
<< snip >>
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#issuecomment-1036593761
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r805323142
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
@ferruzzi you get the error because this line doesn't do what you expect.
You expect `"{{ dag_run.conf['resources_vpc_config'] }}"` to be the Json define in the beginning of the file:
```
{
"subnetIds": ["subnet-12345ab", "subnet-67890cd"],
"endpointPublicAccess": true,
"endpointPrivateAccess": false
}
```
But this is not what is happening. The Jinja engine runs only in templated fields of operators. It does not run in top level python code. So when you execute `json.loads` you actually execute it on the string that you wrote since it wasn't replaced with the Json defined earlier.
If you will run in your python console :
`json.loads("{{ dag_run.conf['resources_vpc_config'] }}")`
You will see it will give you :
`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #21523: Update EKS sample DAGs and docs
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #21523:
URL: https://github.com/apache/airflow/pull/21523#discussion_r805323142
##########
File path: airflow/providers/amazon/aws/example_dags/example_eks_templated.py
##########
@@ -49,50 +44,53 @@
"""
with DAG(
- dag_id='to-publish-manuals-templated',
- default_args={'cluster_name': "{{ dag_run.conf['cluster_name'] }}"},
+ dag_id='example_eks_templated',
schedule_interval=None,
start_date=datetime(2021, 1, 1),
- catchup=False,
- max_active_runs=1,
tags=['example', 'templated'],
+ catchup=False,
# render_template_as_native_obj=True is what converts the Jinja to Python objects, instead of a string.
render_template_as_native_obj=True,
) as dag:
- SUBNETS = os.environ.get('EKS_DEMO_SUBNETS', 'subnet-12345ab subnet-67890cd').split(' ')
- VPC_CONFIG = {
- 'subnetIds': SUBNETS,
- 'endpointPublicAccess': True,
- 'endpointPrivateAccess': False,
- }
+
+ CLUSTER_NAME = "{{ dag_run.conf['cluster_name'] }}"
+ NODEGROUP_NAME = "{{ dag_run.conf['nodegroup_name'] }}"
+ VPC_CONFIG = json.loads("{{ dag_run.conf['resources_vpc_config'] }}")
Review comment:
@ferruzzi you get the error because this line doesn't do what you expect.
You expect `"{{ dag_run.conf['resources_vpc_config'] }}"` to be the Json define in the beginning of the file:
```
{
"subnetIds": ["subnet-12345ab", "subnet-67890cd"],
"endpointPublicAccess": true,
"endpointPrivateAccess": false
}
```
But this is not what is happening. The Jinja engine runs only in templated fields of operators. It does not run in top level python code. So when you execute `json.loads` you actually execute it on the string that you wrote.
If you will run in your python console :
`json.loads("{{ dag_run.conf['resources_vpc_config'] }}")`
You will see it will give you :
`json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org