You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/06 18:27:51 UTC
[GitHub] [airflow] pateash opened a new pull request #19446: #16691 fix passing metadata to dataproc cluster create operator
pateash opened a new pull request #19446:
URL: https://github.com/apache/airflow/pull/19446
closes: #16911
### Add meaningful description above
I am proposing to add a bit more information regarding usage of **DataprocClusterCreateOperator()** as per refactoring in #6371, users find it a bit difficult to generate **CLUSTER_CONFIG**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19446:
URL: https://github.com/apache/airflow/pull/19446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pateash commented on pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
pateash commented on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-999967073
> @pateash I am facing similar issue with **DataprocCreateClusterOperator** as well(Airflow 2+) . Also I see **metadata** field in DataprocCreateClusterOperator is of type Sequence[Tuple[str, str]] but the ClusterGenerator() has type dict for metadata field. This seems to contradict. I see this branch is merged. Also as per my knowledge ClusterGenerator() is deprecated method. Could you let me know the fix for newer Operator
@guptaneha0908 **ClusterGenerator()** has been kept for easy migration, ideally you should have our CLUSTER_CONFIG as dictionary. You can avoid using ClusterGenerator by creating CLUSTER_CONFIG from it once and then storing that dictionary and passing to DataProcClusterCreateOperator going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-968895117
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pateash commented on pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
pateash commented on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-962658912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] guptaneha0908 edited a comment on pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
guptaneha0908 edited a comment on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-998622159
@pateash I am facing similar issue with **DataprocCreateClusterOperator** as well(Airflow 2+) . Also I see **metadata** field in DataprocCreateClusterOperator is of type Sequence[Tuple[str, str]] but the ClusterGenerator() has type dict for metadata field.
This seems to contradict. I see this branch is merged. Also as per my knowledge ClusterGenerator() is deprecated method. Could you let me know the fix for newer Operator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pateash commented on a change in pull request #19446: #16691 fix passing metadata to dataproc cluster create operator
Posted by GitBox <gi...@apache.org>.
pateash commented on a change in pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#discussion_r744285341
##########
File path: airflow/providers/google/cloud/example_dags/example_dataproc.py
##########
@@ -23,13 +23,14 @@
import os
from airflow import models
+from airflow.contrib.operators.dataproc_operator import DataprocClusterCreateOperator
Review comment:
@eladkal @mik-laj
Agree, I will information about older version from here and move it to updating.md
Also this is not only useful while upgrading, In general its much easier to generate CLUSTER_CONFIG from **ClusterGenerator** API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] guptaneha0908 commented on pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
guptaneha0908 commented on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-998622159
@pateash I am facing similar issue with **DataprocCreateClusterOperator** as well(Airflow 2+) . Also I see **metadata** field in DataprocCreateClusterOperator is of type Sequence[Tuple[str, str]] but the ClusterGenerator() has type dict for metadata field.
This seems to contradict. I see this branch is merged. Also as per my knowledge ClusterGenerator() is depredcated method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pateash commented on pull request #19446: #16691 fix passing metadata to dataproc cluster create operator
Posted by GitBox <gi...@apache.org>.
pateash commented on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-962493510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #19446: #16691 fix passing metadata to dataproc cluster create operator
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#discussion_r744169864
##########
File path: airflow/providers/google/cloud/example_dags/example_dataproc.py
##########
@@ -23,13 +23,14 @@
import os
from airflow import models
+from airflow.contrib.operators.dataproc_operator import DataprocClusterCreateOperator
Review comment:
I dont think this is something we want to do.
Our documntation and examples should always support latest airflow version not EOL versions.
If users have trouble with 1.10 they can ask in Disscusion and get support.
It should not be part of the code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pateash removed a comment on pull request #19446: #16691 Providing more information in docs for DataprocCreateCluster operator migration
Posted by GitBox <gi...@apache.org>.
pateash removed a comment on pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#issuecomment-962493510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #19446: #16691 fix passing metadata to dataproc cluster create operator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #19446:
URL: https://github.com/apache/airflow/pull/19446#discussion_r744244021
##########
File path: airflow/providers/google/cloud/example_dags/example_dataproc.py
##########
@@ -23,13 +23,14 @@
import os
from airflow import models
+from airflow.contrib.operators.dataproc_operator import DataprocClusterCreateOperator
Review comment:
Information about the upgrading to the latest version should be to UPDATING.md. https://github.com/apache/airflow/blob/main/UPDATING.md#changes-in-google-provider-package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org