You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/19 11:49:02 UTC
[GitHub] [airflow] Piatachock opened a new pull request, #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Piatachock opened a new pull request, #25158:
URL: https://github.com/apache/airflow/pull/25158
Support following parameters in DataprocCreateClusterOperator:
* `properties`
* `initialization_actions`
* `enable_ui_proxy`
* `security_group_pds`
Support creation of DataProc clusters with no datanodes and S3 instead of HDFS
Add example for direct usage of yandexcloud SDK connection in custom operators
---
Functionality is tested on local AirFlow installation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1194318078
Rebased to account for Werkzeug failures, but likely there are tests to address anyway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1191690538
It's actually explained at the bottom of the error you see in static checks and here are the docs: https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#pre-commit-hooks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25158:
URL: https://github.com/apache/airflow/pull/25158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Piatachock commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
Piatachock commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1190254380
@potiuk hey Jarek, I'm not sure what to do with CI, guide me please.
I've got a dependency version update in this PR, so I updated provider.yaml file. Now CI fails on check on `generated/provider_dependencies.yaml` file: https://github.com/apache/airflow/runs/7428106267?check_suite_focus=true
Based on README in generated/ dir, I'm not sure if I am supposed to modify this file by hand. What would you suggest to do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1191689182
You should simply run pre-commit and commit what has been generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1196806622
Just rebased it (I fixed both problem and merged the fix today). If it works and we merge it - it will get released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1191692451
And feel free to add in the README that you should commit the changes generated by pre-commit (and also you could add link to https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#pre-commit-hooks ). Ideally you are the best person to phrase it correctly as you misunderstood what I was trying to tell there - so if you can improve the README description that woudl be fantastic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Piatachock commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
Piatachock commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1195640508
Not sure if fails are related to changes of this PR;
```
Unknown warnings generated:
[1393](https://github.com/apache/airflow/runs/7520766090?check_suite_focus=true#step:12:1400)
/usr/local/lib/python3.7/site-packages/snowflake/connector/options.py:99:You have an incompatible version of 'pyarrow' installed (6.0.1), please install a version that adheres to: 'pyarrow<8.1.0,>=8.0.0; extra == "pandas"'
...
ERROR! There were 1 warnings generated during the import
```
and
```
[13951](https://github.com/apache/airflow/runs/7520766090?check_suite_focus=true#step:10:13965)
tests/providers/google/common/auth_backend/test_google_openid.py F...... [ 74%]
...
> assert "test@fab.org" == current_user.email
[14171](https://github.com/apache/airflow/runs/7520766090?check_suite_focus=true#step:10:14185)
E AttributeError: 'AnonymousUser' object has no attribute 'email'
```
Do not seem related to this PR for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Piatachock commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
Piatachock commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1191737660
That makes sense, thanks! I guess I was not sure if pre-commit should happen on my side or on CI side.
I'll see if there is other way to phrase it in README
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1195794958
Yep. You modify dependencies so you also upgrade all others, and those errors are indeed happening in main (or were happening) - one of them I already fixed (the warning). The other I know what causes it and (new flask-login) and I am fixing it shortly. Bad luck - you attempted to update dependencies (via provider.yaml) when other dependencies broke main.
:shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Piatachock commented on pull request #25158: YandexCloud provider: Support new Yandex SDK features for DataProc
Posted by GitBox <gi...@apache.org>.
Piatachock commented on PR #25158:
URL: https://github.com/apache/airflow/pull/25158#issuecomment-1196493344
Oh, makes sense, thanks for the explanation. Please let me know if I can help in any way.
On the other hand, we at Yandex would be happy to pack this PR in next providers release; I would be on vacation for next week, but if we can help to make this happen - please let me or @peter-volkov know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org