You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/29 20:40:12 UTC
[GitHub] [airflow] potiuk opened a new pull request #17900: Remove all Deprecation warnings in providers
potiuk opened a new pull request #17900:
URL: https://github.com/apache/airflow/pull/17900
There were a number of deprecation warnings when just importing
certain providers or their examples. Some of them came from still
using 1.10 classes, some from using still `apply_defaults` and
some from badly implemented fallback mechanism to support
backwards compatibility.
This has all been fixed and our documentation generation step
for providers in CI will also fail in case there are some new
DeprecationWarnings generated, which means that they will have
to be fixed before merging.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#discussion_r698356649
##########
File path: tests/always/test_project_structure.py
##########
@@ -163,37 +163,24 @@ class TestGoogleProviderProjectStructure(unittest.TestCase):
}
MISSING_EXAMPLES_FOR_OPERATORS = {
- # Deprecated operator. Ignore it.
+ # Deprecated operators
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceS3ToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceGCSToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor',
- # Base operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_sql.CloudSQLBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator',
'airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocScaleClusterOperator',
- # Base operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocJobBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator',
'airflow.providers.google.cloud.operators.mlengine.MLEngineTrainingCancelJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator',
- # Deprecated operator. Ignore it
Review comment:
Now it is not easy to tell for operators that there are no examples. What do you think to create some variables to break down these operators?
```
BASE_OPERATORS = ['AAA', 'BB']
DEPRECATED_OPERATORS = ['AAA', 'BB']
MISSING_EXAMPLES_FOR_OPERATORS = [
*BASE_OPERATORS, *DEPRECATED_OPERATORS,
'CC', 'DD',
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#discussion_r698374699
##########
File path: tests/always/test_project_structure.py
##########
@@ -163,37 +163,24 @@ class TestGoogleProviderProjectStructure(unittest.TestCase):
}
MISSING_EXAMPLES_FOR_OPERATORS = {
- # Deprecated operator. Ignore it.
+ # Deprecated operators
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceS3ToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceGCSToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor',
- # Base operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_sql.CloudSQLBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator',
'airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocScaleClusterOperator',
- # Base operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocJobBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator',
'airflow.providers.google.cloud.operators.mlengine.MLEngineTrainingCancelJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator',
- # Deprecated operator. Ignore it
Review comment:
Done :) . Good catch. I hope I classifted them correctly :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#issuecomment-908167194
I don’t understand the Big Query stuff; everything else LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#discussion_r698367616
##########
File path: tests/always/test_project_structure.py
##########
@@ -163,37 +163,24 @@ class TestGoogleProviderProjectStructure(unittest.TestCase):
}
MISSING_EXAMPLES_FOR_OPERATORS = {
- # Deprecated operator. Ignore it.
+ # Deprecated operators
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceS3ToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceGCSToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor',
- # Base operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_sql.CloudSQLBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator',
'airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocScaleClusterOperator',
- # Base operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocJobBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator',
'airflow.providers.google.cloud.operators.mlengine.MLEngineTrainingCancelJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator',
- # Deprecated operator. Ignore it
Review comment:
I thought they are all deprecated, to be honest, rather than missing. But if that's the case, then yeah - absolutely we should split them indeed. Give me a moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#issuecomment-908162152
I'd love to merge that one before releasing the new wave of providers #17890, this way all examples and code will be "warning-free". We had many deprecation warnings generated either in examples or when simply importing the providers. This will prevent it from happening also in the future as is checked in the CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#discussion_r698374699
##########
File path: tests/always/test_project_structure.py
##########
@@ -163,37 +163,24 @@ class TestGoogleProviderProjectStructure(unittest.TestCase):
}
MISSING_EXAMPLES_FOR_OPERATORS = {
- # Deprecated operator. Ignore it.
+ # Deprecated operators
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceS3ToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_storage_transfer_service'
'.CloudDataTransferServiceGCSToGCSOperator',
- # Deprecated operator. Ignore it.
'airflow.providers.google.cloud.sensors.gcs.GCSObjectsWtihPrefixExistenceSensor',
- # Base operator. Ignore it.
'airflow.providers.google.cloud.operators.cloud_sql.CloudSQLBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHadoopJobOperator',
'airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocScaleClusterOperator',
- # Base operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocJobBaseOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitSparkSqlJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitHiveJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPigJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.dataproc.DataprocSubmitPySparkJobOperator',
'airflow.providers.google.cloud.operators.mlengine.MLEngineTrainingCancelJobOperator',
- # Deprecated operator. Ignore it
'airflow.providers.google.cloud.operators.mlengine.MLEngineManageModelOperator',
- # Deprecated operator. Ignore it
Review comment:
Done :) . Good catch. I hope I classified them correctly :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17900:
URL: https://github.com/apache/airflow/pull/17900#issuecomment-908187259
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #17900: Remove all deprecation warnings in providers
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #17900:
URL: https://github.com/apache/airflow/pull/17900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org