You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/11 21:28:09 UTC
[GitHub] [airflow] adlersantos opened a new pull request, #24980: Support `service_account_email` attr for Dataflow jobs in Python or Go
adlersantos opened a new pull request, #24980:
URL: https://github.com/apache/airflow/pull/24980
PR #23078 introduced the `service_account` parameter for Dataflow job, but this parameter only works for jobs written in Java. See [Java pipeline options](https://cloud.google.com/dataflow/docs/reference/pipeline-options#java) for Dataflow.
To support Dataflow jobs written in Python or Go, the parameter to use is `service_account_email`. See [Python pipeline options](https://cloud.google.com/dataflow/docs/reference/pipeline-options#python) and [Go pipeline options](https://cloud.google.com/dataflow/docs/reference/pipeline-options#go) for Dataflow.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pabloem commented on a diff in pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
pabloem commented on code in PR #24980:
URL: https://github.com/apache/airflow/pull/24980#discussion_r943747674
##########
airflow/providers/google/cloud/operators/dataflow.py:
##########
@@ -124,7 +124,10 @@ class DataflowConfiguration:
WaitForRun = wait until job finished and the run job.
Supported only by:
:py:class:`~airflow.providers.apache.beam.operators.beam.BeamRunJavaPipelineOperator`
- :param service_account: Run the job as a specific service account, instead of the default GCE robot.
+ :param service_account: Run the Java job as a specific service account, instead of the default Compute Engine service account.
+ See: https://cloud.google.com/dataflow/docs/reference/pipeline-options#java
+ :param service_account_email: Run the Python or Go job as a specific service account for Python or Go, instead of the default Compute Engine service account.
+ See: https://cloud.google.com/dataflow/docs/reference/pipeline-options#python or https://cloud.google.com/dataflow/docs/reference/pipeline-options#go
Review Comment:
I just wonder if it doesn't make sense to put these into a single parameter rather than two separate ones? The documentation is quite clear, so it may not be a big deal - but ideally users only have to care about one parameter?
I understand that complicates the logic downstream, but maybe it's better to complicate downstream than enable users to misconfigure/be confused?
also, sorry about the delay ^^'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
uranusjr commented on code in PR #24980:
URL: https://github.com/apache/airflow/pull/24980#discussion_r958169714
##########
airflow/providers/google/cloud/operators/dataflow.py:
##########
@@ -124,7 +124,10 @@ class DataflowConfiguration:
WaitForRun = wait until job finished and the run job.
Supported only by:
:py:class:`~airflow.providers.apache.beam.operators.beam.BeamRunJavaPipelineOperator`
- :param service_account: Run the job as a specific service account, instead of the default GCE robot.
+ :param service_account: Run the Java job as a specific service account, instead of the default Compute Engine service account.
+ See: https://cloud.google.com/dataflow/docs/reference/pipeline-options#java
+ :param service_account_email: Run the Python or Go job as a specific service account for Python or Go, instead of the default Compute Engine service account.
+ See: https://cloud.google.com/dataflow/docs/reference/pipeline-options#python or https://cloud.google.com/dataflow/docs/reference/pipeline-options#go
Review Comment:
How well does the API handle unknown keys? If unknown keys are simply ignored, we could probably just always set both keys. Otherwise the implementation might be pretty complicated and not worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
URL: https://github.com/apache/airflow/pull/24980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #24980:
URL: https://github.com/apache/airflow/pull/24980#issuecomment-1204368223
Can you please fix static checks?
```
Run flake8.............................................................................Failed
- hook id: run-flake8
- exit code: 1
airflow/providers/google/cloud/operators/dataflow.py:127:111: E501 line too long (130 > 110 characters)
airflow/providers/google/cloud/operators/dataflow.py:129:111: E501 line too long (161 > 110 characters)
airflow/providers/google/cloud/operators/dataflow.py:130:111: E501 line too long (157 > 110 characters)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
mik-laj commented on PR #24980:
URL: https://github.com/apache/airflow/pull/24980#issuecomment-1186539753
@pabloem Can you look at it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24980:
URL: https://github.com/apache/airflow/pull/24980#issuecomment-1279607840
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pabloem commented on a diff in pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
pabloem commented on code in PR #24980:
URL: https://github.com/apache/airflow/pull/24980#discussion_r1000037185
##########
airflow/providers/google/cloud/operators/dataflow.py:
##########
@@ -124,7 +124,10 @@ class DataflowConfiguration:
WaitForRun = wait until job finished and the run job.
Supported only by:
:py:class:`~airflow.providers.apache.beam.operators.beam.BeamRunJavaPipelineOperator`
- :param service_account: Run the job as a specific service account, instead of the default GCE robot.
+ :param service_account: Run the Java job as a specific service account, instead of the default Compute Engine service account.
+ See: https://cloud.google.com/dataflow/docs/reference/pipeline-options#java
+ :param service_account_email: Run the Python or Go job as a specific service account for Python or Go, instead of the default Compute Engine service account.
+ See: https://cloud.google.com/dataflow/docs/reference/pipeline-options#python or https://cloud.google.com/dataflow/docs/reference/pipeline-options#go
Review Comment:
once again my apologies for missing this - I am not sure about how the API deals with this. If not, I am happy to move forward with the way it is...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #24980: Support `service_account_email` attr for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #24980:
URL: https://github.com/apache/airflow/pull/24980#issuecomment-1180887978
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24980: Support `service_account_email` pipeline option for Dataflow jobs in Python or Go
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24980:
URL: https://github.com/apache/airflow/pull/24980#issuecomment-1223965821
Any comments/feedbcack @adlersantos ? Are you still working on it (it needs at least reabase and likely the static checks fixed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org