You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/08 03:31:26 UTC
[GitHub] [airflow] otourzan opened a new pull request #12907: Add regional support to the workflow template methods.
otourzan opened a new pull request #12907:
URL: https://github.com/apache/airflow/pull/12907
Workflow templates of GCP can be regional or global. In case of
regional the GCP API endpoint rpc url should match to the same
region.
In case of global templates needed to pass 'global' as region.
It is not used for endpoint address but needed to as part of
template path.
ISSUE: #12804
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#issuecomment-749165383
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] otourzan commented on a change in pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
otourzan commented on a change in pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#discussion_r540973336
##########
File path: airflow/providers/google/cloud/hooks/dataproc.py
##########
@@ -218,11 +217,14 @@ def get_cluster_client(self, location: Optional[str] = None) -> ClusterControlle
credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
)
- @cached_property
- def get_template_client(self) -> WorkflowTemplateServiceClient:
+ def get_template_client(self, location: Optional[str] = None) -> WorkflowTemplateServiceClient:
"""Returns WorkflowTemplateServiceClient."""
+ client_options = None
+ if location and location != 'global':
Review comment:
Yes, user should provide proper configuration. If they pass 'global' as a valid location, the endpoint needs to be global one (without region prefix: 'dataproc.googleapis.com') as there is endpoint like 'global-dataproc.googleapis.com'. Not sure if I got your point here.
I joined the Airflow slack and will be happy to help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#issuecomment-743221255
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#discussion_r538836953
##########
File path: airflow/providers/google/cloud/hooks/dataproc.py
##########
@@ -218,11 +217,14 @@ def get_cluster_client(self, location: Optional[str] = None) -> ClusterControlle
credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
)
- @cached_property
- def get_template_client(self) -> WorkflowTemplateServiceClient:
+ def get_template_client(self, location: Optional[str] = None) -> WorkflowTemplateServiceClient:
"""Returns WorkflowTemplateServiceClient."""
+ client_options = None
+ if location and location != 'global':
Review comment:
Btw. I see that you are Google related, are you on Airflow slack? My colleague @michalslowikowski00 is refactoring Dataproc operators to use dataproc python library 2.0 and having someone to consult would be lovely π
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] otourzan commented on a change in pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
otourzan commented on a change in pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#discussion_r540973336
##########
File path: airflow/providers/google/cloud/hooks/dataproc.py
##########
@@ -218,11 +217,14 @@ def get_cluster_client(self, location: Optional[str] = None) -> ClusterControlle
credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
)
- @cached_property
- def get_template_client(self) -> WorkflowTemplateServiceClient:
+ def get_template_client(self, location: Optional[str] = None) -> WorkflowTemplateServiceClient:
"""Returns WorkflowTemplateServiceClient."""
+ client_options = None
+ if location and location != 'global':
Review comment:
Yes, user should provide proper configuration. If they pass 'global' as a valid location, the endpoint needs to be global one (without region prefix: 'dataproc.googleapis.com') as there is *no* endpoint like 'global-dataproc.googleapis.com'. Not sure if I got your point here.
I joined the Airflow slack and will be happy to help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#issuecomment-740351497
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itβs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better π.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #12907:
URL: https://github.com/apache/airflow/pull/12907
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] otourzan commented on a change in pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
otourzan commented on a change in pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#discussion_r538308549
##########
File path: airflow/providers/google/cloud/hooks/dataproc.py
##########
@@ -218,11 +217,14 @@ def get_cluster_client(self, location: Optional[str] = None) -> ClusterControlle
credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
)
- @cached_property
- def get_template_client(self) -> WorkflowTemplateServiceClient:
+ def get_template_client(self, location: Optional[str] = None) -> WorkflowTemplateServiceClient:
"""Returns WorkflowTemplateServiceClient."""
+ client_options = None
+ if location and location != 'global':
Review comment:
If template is in global namespace, region (location) parameter needed to be set to 'global' as required for template path [1], [2].
[1]: https://github.com/apache/airflow/blob/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5/airflow/providers/google/cloud/hooks/dataproc.py#L645
[2]: https://github.com/googleapis/python-dataproc/blob/7ec92b71be9c1d0d305421bb1b1dce5d92377bba/google/cloud/dataproc_v1beta2/services/workflow_template_service/client.py#L155
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#discussion_r538835043
##########
File path: airflow/providers/google/cloud/hooks/dataproc.py
##########
@@ -218,11 +217,14 @@ def get_cluster_client(self, location: Optional[str] = None) -> ClusterControlle
credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
)
- @cached_property
- def get_template_client(self) -> WorkflowTemplateServiceClient:
+ def get_template_client(self, location: Optional[str] = None) -> WorkflowTemplateServiceClient:
"""Returns WorkflowTemplateServiceClient."""
+ client_options = None
+ if location and location != 'global':
Review comment:
@otourzan in first link users have to provide the value for `location` so it's up to them to provide proper configuration. Or do I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #12907: Add regional support to the workflow template methods.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #12907:
URL: https://github.com/apache/airflow/pull/12907#discussion_r538287734
##########
File path: airflow/providers/google/cloud/hooks/dataproc.py
##########
@@ -218,11 +217,14 @@ def get_cluster_client(self, location: Optional[str] = None) -> ClusterControlle
credentials=self._get_credentials(), client_info=self.client_info, client_options=client_options
)
- @cached_property
- def get_template_client(self) -> WorkflowTemplateServiceClient:
+ def get_template_client(self, location: Optional[str] = None) -> WorkflowTemplateServiceClient:
"""Returns WorkflowTemplateServiceClient."""
+ client_options = None
+ if location and location != 'global':
Review comment:
Do we need the case for `global`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org