You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/11 23:44:36 UTC
[GitHub] [airflow] potiuk opened a new issue, #23659: Status of testing Providers that were prepared on May 12, 2022
potiuk opened a new issue, #23659:
URL: https://github.com/apache/airflow/issues/23659
### Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
## Provider [amazon: 3.4.0rc1](https://pypi.org/project/apache-airflow-providers-amazon/3.4.0rc1)
- [ ] [Add Quicksight create ingestion Hook and Operator (#21863)](https://github.com/apache/airflow/pull/21863): @hsrocks
- [ ] [Add default 'aws_conn_id' to SageMaker Operators (#23515)](https://github.com/apache/airflow/pull/23515): @hsrocks
- [ ] [Add `S3CreateObjectOperator` (#22758)](https://github.com/apache/airflow/pull/22758): @vincbeck
- [ ] [Add `RedshiftDeleteClusterOperator` support (#23563)](https://github.com/apache/airflow/pull/23563): @pankajastro
- [ ] [Fix conn close error on retrieving log events (#23470)](https://github.com/apache/airflow/pull/23470): @thinhnd2104
- [ ] [Fix LocalFilesystemToS3Operator and S3CreateObjectOperator to support full s3:// style keys (#23180)](https://github.com/apache/airflow/pull/23180): @vincbeck
- [ ] [Fix attempting to reattach in `ECSOperator` (#23370)](https://github.com/apache/airflow/pull/23370): @fshehadeh
- [ ] [Fix doc build failure on main (#23240)](https://github.com/apache/airflow/pull/23240): @uranusjr
- [ ] [`S3Hook`: fix `load_bytes` docstring (#23182)](https://github.com/apache/airflow/pull/23182): @jasonwashburn
- [ ] [Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor` (#22737)](https://github.com/apache/airflow/pull/22737): @vincbeck
- [ ] [Allow back script_location in Glue to be None (#23357)](https://github.com/apache/airflow/pull/23357): @potiuk
nikolas
## Provider [apache.hive: 2.3.3rc1](https://pypi.org/project/apache-airflow-providers-apache-hive/2.3.3rc1)
- [ ] [Fix HiveToMySqlOperator's wrong docstring (#23316)](https://github.com/apache/airflow/pull/23316): @sekikn
## Provider [cncf.kubernetes: 4.0.2rc1](https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/4.0.2rc1)
- [ ] [Fix: Exception when parsing log #20966 (#23301)](https://github.com/apache/airflow/pull/23301): @jakubno
- [ ] [Fixed Kubernetes Operator large xcom content Defect (#23490)](https://github.com/apache/airflow/pull/23490): @rahulgoyal2987
- [ ] [Clarify `reattach_on_restart` behavior (#23377)](https://github.com/apache/airflow/pull/23377): @jedcunningham
## Provider [databricks: 2.7.0rc1](https://pypi.org/project/apache-airflow-providers-databricks/2.7.0rc1)
- [ ] [Further improvement of Databricks Jobs operators (mostly docs) (#23199)](https://github.com/apache/airflow/pull/23199): @alexott
## Provider [docker: 2.7.0rc1](https://pypi.org/project/apache-airflow-providers-docker/2.7.0rc1)
- [ ] [Add `device_requests` parameter to `DockerOperator` (#23554)](https://github.com/apache/airflow/pull/23554): @eladkal
- [ ] [Fix new MyPy errors in main (#22884)](https://github.com/apache/airflow/pull/22884): @potiuk
## Provider [google: 7.0.0rc1](https://pypi.org/project/apache-airflow-providers-google/7.0.0rc1)
- [ ] [Remove deprecated parameters from PubSub operators (#23261)](https://github.com/apache/airflow/pull/23261): @eladkal
- [ ] [Upgrade to support Google Ads v10 (#22965)](https://github.com/apache/airflow/pull/22965): @anthonyjgraff
- [ ] [Dataproc: Remove default value of `region` (#23350)](https://github.com/apache/airflow/pull/23350): @eladkal
- [ ] [`DatastoreHook`: Remove `datastore_conn_id` (#23323)](https://github.com/apache/airflow/pull/23323): @eladkal
- [ ] [`CloudBuildCreateBuildOperator`: Remove deprecated `body` parameter (#23263)](https://github.com/apache/airflow/pull/23263): @eladkal
- [ ] [Bigtable: Remove `replica_cluster_id`, `replica_cluster_zone` (#23251)](https://github.com/apache/airflow/pull/23251): @eladkal
- [ ] [Remove deprecated `params` from google operators (#23230)](https://github.com/apache/airflow/pull/23230): @eladkal
- [ ] [`GoogleDriveToGCSOperator`: Remove `destination_bucket`, `destination_object` (#23072)](https://github.com/apache/airflow/pull/23072): @eladkal
- [ ] [Remove `GCSObjectsWtihPrefixExistenceSensor` (#23050)](https://github.com/apache/airflow/pull/23050): @eladkal
- [ ] [`PubSubPullSensor`: Remove `project` and `return_immediately` (#23231)](https://github.com/apache/airflow/pull/23231): @eladkal
- [ ] [Dataproc : remove `location` in favor of `region` (#23250)](https://github.com/apache/airflow/pull/23250): @eladkal
- [ ] [Datastore: Remove `xcom_push` (#23252)](https://github.com/apache/airflow/pull/23252): @eladkal
- [ ] [`S3ToGCSOperator`: Remove `dest_gcs_conn_id` (#23348)](https://github.com/apache/airflow/pull/23348): @eladkal
- [ ] [Remove deprecated parameters from BigQueryHook (#23269)](https://github.com/apache/airflow/pull/23269): @eladkal
- [ ] [`DataprocHook`: Remove deprecated function `submit` (#23389)](https://github.com/apache/airflow/pull/23389): @eladkal
- [ ] [[FEATURE] google provider - BigQueryInsertJobOperator log query (#23648)](https://github.com/apache/airflow/pull/23648): @raphaelauv
- [ ] [[FEATURE] google provider - split GkeStartPodOperator execute (#23518)](https://github.com/apache/airflow/pull/23518): @raphaelauv
- [ ] [Add exportContext.offload flag to CLOUD_SQL_EXPORT_VALIDATION. (#23614)](https://github.com/apache/airflow/pull/23614): @ishiis
- [ ] [Create links for Biqtable operators (#23164)](https://github.com/apache/airflow/pull/23164): @MaksYermak
- [ ] [implements #22859 - Add .sql as templatable extension (#22920)](https://github.com/apache/airflow/pull/22920): @LennyKLB
- [ ] [`GCSFileTransformOperator`: New templated fields `source_object`, `destination_object` (#23328)](https://github.com/apache/airflow/pull/23328): @eladkal
- [ ] [Fix `PostgresToGCSOperator` does not allow nested JSON (#23063)](https://github.com/apache/airflow/pull/23063): @pierrejeambrun
- [ ] [Fix GCSToGCSOperator ignores replace parameter when there is no wildcard (#23340)](https://github.com/apache/airflow/pull/23340): @gitstart-airflow
- [ ] [Fix broken CloudSQLProxyRunner download URLs (#23299)](https://github.com/apache/airflow/pull/23299): @jackwotherspoon
- [ ] [`LookerStartPdtBuildOperator`, `LookerCheckPdtBuildSensor` : fix empty materialization id handling (#23025)](https://github.com/apache/airflow/pull/23025): @alekseiloginov
- [ ] [Change ComputeSSH to throw provider import error instead paramiko (#23035)](https://github.com/apache/airflow/pull/23035): @potiuk
- [ ] [Fix cancel_on_kill after execution timeout for DataprocSubmitJobOperator (#22955)](https://github.com/apache/airflow/pull/22955): @tauzen
- [ ] [Fix BigQueryGetDataOperator Response Serialisation (#22936)](https://github.com/apache/airflow/pull/22936): @pankajastro
- [ ] [MSSQLToGCSOperator fails: datetime is not JSON Serializable (#22882)](https://github.com/apache/airflow/pull/22882): @pierrejeambrun
## Provider [microsoft.azure: 3.9.0rc1](https://pypi.org/project/apache-airflow-providers-microsoft-azure/3.9.0rc1)
- [ ] [Use DefaultAzureCredential instead of ManagedIdentityCredential in azure wasb hook (#23394)](https://github.com/apache/airflow/pull/23394): @snjypl
- [ ] [Replace usage of `DummyOperator` with `EmptyOperator` (#22974)](https://github.com/apache/airflow/pull/22974): @eladkal
## Provider [opsgenie: 3.1.0rc1](https://pypi.org/project/apache-airflow-providers-opsgenie/3.1.0rc1)
- [ ] [Add `OpsgenieDeleteAlertOperator` (#23405)](https://github.com/apache/airflow/pull/23405): @eladkal
- [ ] [Opsgenie: Fix `close_alert` to properly send `kwargs` (#23442)](https://github.com/apache/airflow/pull/23442): @bperson
## Provider [presto: 2.2.1rc1](https://pypi.org/project/apache-airflow-providers-presto/2.2.1rc1)
- [ ] [Make presto and trino compatible with airflow 2.1 (#23061)](https://github.com/apache/airflow/pull/23061): @pingzh
## Provider [salesforce: 3.4.4rc1](https://pypi.org/project/apache-airflow-providers-salesforce/3.4.4rc1)
- [ ] [Organize Tableau classes (#23353)](https://github.com/apache/airflow/pull/23353): @eladkal
## Provider [snowflake: 2.7.0rc1](https://pypi.org/project/apache-airflow-providers-snowflake/2.7.0rc1)
- [ ] [Allow multiline text in private key field for Snowflake (#23066)](https://github.com/apache/airflow/pull/23066): @mik-laj
## Provider [ssh: 2.4.4rc1](https://pypi.org/project/apache-airflow-providers-ssh/2.4.4rc1)
- [ ] [Add exception to catch single line private keys (#23043)](https://github.com/apache/airflow/pull/23043): @nsAstro
## Provider [tableau: 2.1.8rc1](https://pypi.org/project/apache-airflow-providers-tableau/2.1.8rc1)
- [ ] [Organize Tableau classes (#23353)](https://github.com/apache/airflow/pull/23353): @eladkal
## Provider [trino: 2.3.0rc1](https://pypi.org/project/apache-airflow-providers-trino/2.3.0rc1)
- [ ] [TrinoHook add authentication via JWT token and Impersonation (#23116)](https://github.com/apache/airflow/pull/23116): @pragyak412
- [ ] [Make presto and trino compatible with airflow 2.1 (#23061)](https://github.com/apache/airflow/pull/23061): @pingzh
The guidelines on how to test providers can be found in
[Verify providers by contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors)
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
pankajastro commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1127971779
Tested https://github.com/apache/airflow/pull/23563 and https://github.com/apache/airflow/pull/22936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
vincbeck commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1127862536
#22758, #23180 and #22737 tested and works as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1127570531
Last chance for some more testing :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1124890910
Tested Databricks provider:
* #22886 - isn't listed in the Changelog explicitly because it was merge with squash
* #23199 - primarily documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hsrocks commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
hsrocks commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1127712793
#21863 and #23515 -> Tested and Works fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #23659: Status of testing Providers that were prepared on May 12, 2022
URL: https://github.com/apache/airflow/issues/23659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1124896841
> * [Databricks SQL operators are now Python 3.10 compatible #22886](https://github.com/apache/airflow/pull/22886) - isn't listed in the Changelog explicitly because it was merge with squash
Yeah. those are mentioned in the changelog actually but missing from the "Status" here - thanks for checking it too :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] snjypl commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
snjypl commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1126310497
https://github.com/apache/airflow/pull/23394 is tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thinhnd2104 commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
thinhnd2104 commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1125648826
#23470 -> work on my production environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1128147309
Thanks everyone! The providers are released now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ishiis commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
ishiis commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1125647244
#23614 is tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] anthonyjgraff commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
anthonyjgraff commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1126977028
#22965 -> tested DAGs in development environment against v10 of Google Ads API - all worked as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] raphaelauv commented on issue #23659: Status of testing Providers that were prepared on May 12, 2022
Posted by GitBox <gi...@apache.org>.
raphaelauv commented on issue #23659:
URL: https://github.com/apache/airflow/issues/23659#issuecomment-1124405054
#23518 -> WORK
```log
[2022-05-12, 00:22:25 UTC] {base.py:68} INFO - Using connection ID 'google_cloud_default' for task execution.
[2022-05-12, 00:22:27 UTC] {process_utils.py:165} INFO - Executing cmd: gcloud container clusters get-credentials gke-XXX-XXX --project XXXXX --zone europe-west1
[2022-05-12, 00:22:27 UTC] {process_utils.py:169} INFO - Output:
[2022-05-12, 00:22:28 UTC] {process_utils.py:173} INFO - WARNING: Accessing a Kubernetes Engine cluster requires the kubernetes commandline
[2022-05-12, 00:22:28 UTC] {process_utils.py:173} INFO - client [kubectl]. To install, run
[2022-05-12, 00:22:28 UTC] {process_utils.py:173} INFO - $ gcloud components install kubectl
[2022-05-12, 00:22:28 UTC] {process_utils.py:173} INFO -
[2022-05-12, 00:22:28 UTC] {process_utils.py:173} INFO - Fetching cluster endpoint and auth data.
[2022-05-12, 00:22:28 UTC] {process_utils.py:173} INFO - kubeconfig entry generated for gke-XXX-XXX.
[2022-05-12, 00:22:28 UTC] {kubernetes_pod.py:540} INFO - Creating pod airflow-test-pod-33e0676e00ac49c19f699f1c3916deae with labels: {'dag_id': 'kubernetes_dag', 'task_id': 'task-one', 'run_id': 'scheduled__2022-05-10T0000000000-e98a7571d', 'kubernetes_pod_operator': 'True', 'try_number': '3'}
[2022-05-12, 00:22:29 UTC] {pod_manager.py:178} WARNING - Pod not yet started: airflow-test-pod-33e0676e00ac49c19f699f1c3916deae
[2022-05-12, 00:22:31 UTC] {pod_manager.py:178} WARNING - Pod not yet started: airflow-test-pod-33e0676e00ac49c19f699f1c3916deae
[2022-05-12, 00:22:32 UTC] {pod_manager.py:178} WARNING - Pod not yet started: airflow-test-pod-33e0676e00ac49c19f699f1c3916deae
[2022-05-12, 00:22:33 UTC] {pod_manager.py:225} INFO - toto
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org