You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/15 00:49:14 UTC
[GitHub] [airflow] potiuk opened a new issue, #27674: Status of testing Providers that were prepared on November 15, 2022
potiuk opened a new issue, #27674:
URL: https://github.com/apache/airflow/issues/27674
### Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
## Provider [alibaba: 2.2.0rc1](https://pypi.org/project/apache-airflow-providers-alibaba/2.2.0rc1)
- [ ] [Use log.exception where more economical than log.error (#27517)](https://github.com/apache/airflow/pull/27517): @dstandish
- [ ] [Replace urlparse with urlsplit (#27389)](https://github.com/apache/airflow/pull/27389): @WestonKing-Leatham
## Provider [amazon: 6.1.0rc1](https://pypi.org/project/apache-airflow-providers-amazon/6.1.0rc1)
- [ ] [Replace urlparse with urlsplit (#27389)](https://github.com/apache/airflow/pull/27389): @WestonKing-Leatham
- [ ] [Add info about JSON Connection format for AWS SSM Parameter Store Secrets Backend (#27134)](https://github.com/apache/airflow/pull/27134): @Taragolis
- [ ] [Add default name to EMR Serverless jobs (#27458)](https://github.com/apache/airflow/pull/27458): @syedahsn
- [ ] [Adding `preserve_file_name` param to `S3Hook.download_file` method (#26886)](https://github.com/apache/airflow/pull/26886): @alexkruc
- [ ] [Add GlacierUploadArchiveOperator (#26652)](https://github.com/apache/airflow/pull/26652): @pankajastro
- [ ] [Add RdsStopDbOperator and RdsStartDbOperator (#27076)](https://github.com/apache/airflow/pull/27076): @hankehly
- [ ] [`GoogleApiToS3Operator` : add `gcp_conn_id` to template fields (#27017)](https://github.com/apache/airflow/pull/27017): @syedahsn
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Add information about Amazon Elastic MapReduce Connection (#26687)](https://github.com/apache/airflow/pull/26687): @Taragolis
- [ ] [Add Batch Operator template fields (#26805)](https://github.com/apache/airflow/pull/26805): @ferruzzi
- [ ] [Improve testing AWS Connection response (#26953)](https://github.com/apache/airflow/pull/26953): @Taragolis
- [ ] [SagemakerProcessingOperator stopped honoring `existing_jobs_found` (#27456)](https://github.com/apache/airflow/pull/27456): @ferruzzi
- [ ] [CloudWatch task handler doesn't fall back to local logs when Amazon CloudWatch logs aren't found (#27564)](https://github.com/apache/airflow/pull/27564): @hankehly
- [ ] [Fix backwards compatibility for RedshiftSQLOperator (#27602)](https://github.com/apache/airflow/pull/27602): @syedahsn
- [ ] [Fix typo in redshift sql hook `get_ui_field_behaviour` (#27533)](https://github.com/apache/airflow/pull/27533): @pankajastro
- [ ] [Fix example_emr_serverless system test (#27149)](https://github.com/apache/airflow/pull/27149): @vincbeck
- [ ] [[Docs] Fix duplicate param in docstring RedshiftSQLHook `get_table_primary_key` method (#27330)](https://github.com/apache/airflow/pull/27330): @pankajastro
- [ ] [Adds s3_key_prefix to template fields (#27207)](https://github.com/apache/airflow/pull/27207): @mfjackson
- [ ] [Fix assume role if user explicit set credentials (#26946)](https://github.com/apache/airflow/pull/26946): @Taragolis
- [ ] [Fix failure state in waiter call for EmrServerlessStartJobOperator (#26853)](https://github.com/apache/airflow/pull/26853): @syedahsn
- [ ] [Fix a bunch of deprecation warnings AWS tests (#26857)](https://github.com/apache/airflow/pull/26857): @uranusjr
- [ ] [Fix null strings bug in SqlToS3Operator in non parquet formats (#26676)](https://github.com/apache/airflow/pull/26676): @harryplumer
- [ ] [Sagemaker hook: remove extra call at the end when waiting for completion (#27551)](https://github.com/apache/airflow/pull/27551): @vandonr-amz
- [ ] [ECS Buglette (#26921)](https://github.com/apache/airflow/pull/26921): @ferruzzi
- [ ] [Avoid circular imports in AWS Secrets Backends if obtain secrets from config (#26784)](https://github.com/apache/airflow/pull/26784): @Taragolis
## Provider [apache.beam: 4.1.0rc1](https://pypi.org/project/apache-airflow-providers-apache-beam/4.1.0rc1)
- [ ] [Add backward compatibility with old versions of Apache Beam (#27263)](https://github.com/apache/airflow/pull/27263): @mik-laj
## Provider [apache.drill: 2.3.0rc1](https://pypi.org/project/apache-airflow-providers-apache-drill/2.3.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [apache.druid: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-apache-druid/3.3.0rc1)
- [ ] [BugFix - Druid Airflow Exception to about content (#27174)](https://github.com/apache/airflow/pull/27174): @emincanoguz11
## Provider [apache.hive: 4.1.0rc1](https://pypi.org/project/apache-airflow-providers-apache-hive/4.1.0rc1)
- [ ] [Filter out invalid schemas in Hive hook (#27647)](https://github.com/apache/airflow/pull/27647): @potiuk
## Provider [apache.livy: 3.2.0rc1](https://pypi.org/project/apache-airflow-providers-apache-livy/3.2.0rc1)
- [ ] [Add template to Livy operator documentation (#27404)](https://github.com/apache/airflow/pull/27404): @bdsoha
- [ ] [Add Spark's `appId` to xcom output (#27376)](https://github.com/apache/airflow/pull/27376): @bdsoha
- [ ] [Add template field renderer to LivyOperator (#27321)](https://github.com/apache/airflow/pull/27321): @bdsoha
## Provider [apache.pig: 4.0.0rc1](https://pypi.org/project/apache-airflow-providers-apache-pig/4.0.0rc1)
- [ ] [Pig cli properties cannot be passed by connection extra (#27644)](https://github.com/apache/airflow/pull/27644): @potiuk
## Provider [apache.pinot: 4.0.0rc1](https://pypi.org/project/apache-airflow-providers-apache-pinot/4.0.0rc1)
- [ ] [The pinot-admin.sh command is now hard-coded. (#27641)](https://github.com/apache/airflow/pull/27641): @potiuk
- [ ] [Bump pinotdb version (#27201)](https://github.com/apache/airflow/pull/27201): @eladkal
## Provider [apache.spark: 4.0.0rc1](https://pypi.org/project/apache-airflow-providers-apache-spark/4.0.0rc1)
- [ ] [Remove custom spark home and custom binarires for spark (#27646)](https://github.com/apache/airflow/pull/27646): @potiuk
## Provider [arangodb: 2.1.0rc1](https://pypi.org/project/apache-airflow-providers-arangodb/2.1.0rc1)
- [ ] [Fix links to sources for examples (#24386)](https://github.com/apache/airflow/pull/24386): @potiuk
## Provider [asana: 3.0.0rc1](https://pypi.org/project/apache-airflow-providers-asana/3.0.0rc1)
- [ ] [Allow and prefer non-prefixed extra fields for AsanaHook (#27043)](https://github.com/apache/airflow/pull/27043): @dstandish
## Provider [cncf.kubernetes: 5.0.0rc3](https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/5.0.0rc3)
- [ ] [Remove deprecated backcompat objects for KPO (#27518)](https://github.com/apache/airflow/pull/27518): @dstandish
- [ ] [Remove support for node_selectors param in KPO (#27515)](https://github.com/apache/airflow/pull/27515): @dstandish
- [ ] [Remove unused backcompat method in k8s hook (#27490)](https://github.com/apache/airflow/pull/27490): @dstandish
- [ ] [Drop support for providing ``resource`` as dict in ``KubernetesPodOperator`` (#27197)](https://github.com/apache/airflow/pull/27197): @eladkal
- [ ] [Deprecate use of core kube_client in PodManager (#26848)](https://github.com/apache/airflow/pull/26848): @dstandish
- [ ] [Don't consider airflow core conf for KPO (#26849)](https://github.com/apache/airflow/pull/26849): @dstandish
- [ ] [Use log.exception where more economical than log.error (#27517)](https://github.com/apache/airflow/pull/27517): @dstandish
- [ ] [Add container_resources as KubernetesPodOperator templatable (#27457)](https://github.com/apache/airflow/pull/27457): @alanatlemba
- [ ] [Add deprecation warning re unset namespace in k8s hook (#27202)](https://github.com/apache/airflow/pull/27202): @dstandish
- [ ] [add container_name option for SparkKubernetesSensor (#26560)](https://github.com/apache/airflow/pull/26560): @hanna-liashchuk
- [ ] [Allow xcom sidecar container image to be configurable (#26766)](https://github.com/apache/airflow/pull/26766): @bdsoha
- [ ] [Improve task_id to pod name conversion (#27524)](https://github.com/apache/airflow/pull/27524): @dstandish
- [ ] [Make pod name optional in KubernetesPodOperator (#27120)](https://github.com/apache/airflow/pull/27120): @dstandish
- [ ] [Make namespace optional for KPO (#27116)](https://github.com/apache/airflow/pull/27116): @dstandish
- [ ] [Enable template rendering for env_vars field for the @task.kubernetes decorator (#27433)](https://github.com/apache/airflow/pull/27433): @fletchjeff
- [ ] [Fix KubernetesHook failed on an attribute metadata:name absence (#25787)](https://github.com/apache/airflow/pull/25787): @morooshka
- [ ] [Fix log message for kubernetes hooks (#26999)](https://github.com/apache/airflow/pull/26999): @FingerLiu
- [ ] [Remove extra__kubernetes__ prefix from k8s hook extras (#27021)](https://github.com/apache/airflow/pull/27021): @dstandish
- [ ] [KPO should use hook's get namespace method to get namespace (#27516)](https://github.com/apache/airflow/pull/27516): @dstandish
## Provider [common.sql: 1.3.0rc1](https://pypi.org/project/apache-airflow-providers-common-sql/1.3.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first (#26944)](https://github.com/apache/airflow/pull/26944): @kazanzhy
- [ ] [DbApiHook consistent insert_rows logging (#26758)](https://github.com/apache/airflow/pull/26758): @curlup
- [ ] [Common sql bugfixes and improvements (#26761)](https://github.com/apache/airflow/pull/26761): @denimalpaca
- [ ] [Use unused SQLCheckOperator.parameters in SQLCheckOperator.execute. (#27599)](https://github.com/apache/airflow/pull/27599): @wjmolina
## Provider [databricks: 3.4.0rc1](https://pypi.org/project/apache-airflow-providers-databricks/3.4.0rc1)
- [ ] [Replace urlparse with urlsplit (#27389)](https://github.com/apache/airflow/pull/27389): @WestonKing-Leatham
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Use new job search API for triggering Databricks job by name (#27446)](https://github.com/apache/airflow/pull/27446): @alexott
## Provider [docker: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-docker/3.3.0rc1)
- [ ] [Add ipc_mode for DockerOperator (#27553)](https://github.com/apache/airflow/pull/27553): @mrk-andreev
- [ ] [Add env-file parameter to Docker Operator (#26951)](https://github.com/apache/airflow/pull/26951): @sshivprasad
## Provider [exasol: 4.1.0rc1](https://pypi.org/project/apache-airflow-providers-exasol/4.1.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first (#26944)](https://github.com/apache/airflow/pull/26944): @kazanzhy
## Provider [google: 8.5.0rc1](https://pypi.org/project/apache-airflow-providers-google/8.5.0rc1)
- [ ] [Rename hook bigquery function `_bq_cast` to `bq_cast` (#27543)](https://github.com/apache/airflow/pull/27543): @pankajastro
- [ ] [Use non-deprecated method for on_kill in BigQueryHook (#27547)](https://github.com/apache/airflow/pull/27547): @dstandish
- [ ] [Typecast bigquery job response col value (#27236)](https://github.com/apache/airflow/pull/27236): @pankajastro
- [ ] [Remove <2 limit on google-cloud-storage (#26922)](https://github.com/apache/airflow/pull/26922): @r-richmond
- [ ] [Add backward compatibility with old versions of Apache Beam (#27263)](https://github.com/apache/airflow/pull/27263): @mik-laj
- [ ] [Add deferrable mode to GCPToBigQueryOperator + tests (#27052)](https://github.com/apache/airflow/pull/27052): @VladaZakharova
- [ ] [Add system tests for Vertex AI operators in new approach (#27053)](https://github.com/apache/airflow/pull/27053): @VladaZakharova
- [ ] [Dataform operators, links, update system tests and docs (#27144)](https://github.com/apache/airflow/pull/27144): @MrGeorgeOwl
- [ ] [Allow values in WorkflowsCreateExecutionOperator execution argument to be dicts (#27361)](https://github.com/apache/airflow/pull/27361): @rkarish
- [ ] [DataflowStopJobOperator Operator (#27033)](https://github.com/apache/airflow/pull/27033): @Voldurk
- [ ] [Allow for the overriding of stringify_dict for json/jsonb column data… (#26876)](https://github.com/apache/airflow/pull/26876): @sleepy-tiger
- [ ] [Add new Compute Engine Operators and fix system tests (#25608)](https://github.com/apache/airflow/pull/25608): @VladaZakharova
- [ ] [Allow and prefer non-prefixed extra fields for dataprep hook (#27039)](https://github.com/apache/airflow/pull/27039): @dstandish
- [ ] [Common sql bugfixes and improvements (#26761)](https://github.com/apache/airflow/pull/26761): @denimalpaca
- [ ] [Update google hooks to prefer non-prefixed extra fields (#27023)](https://github.com/apache/airflow/pull/27023): @dstandish
- [ ] [Fix delay in Dataproc CreateBatch operator (#26126)](https://github.com/apache/airflow/pull/26126): @VladaZakharova
- [ ] [Remove unnecessary newlines around single arg in signature (#27525)](https://github.com/apache/airflow/pull/27525): @dstandish
- [ ] [set project_id and location when canceling BigQuery job (#27521)](https://github.com/apache/airflow/pull/27521): @lidalei
- [ ] [use `id` key to retrieve the dataflow job_id (#27336)](https://github.com/apache/airflow/pull/27336): @dejii
- [ ] [Make GSheetsHook return an empty list when there are no values (#27261)](https://github.com/apache/airflow/pull/27261): @alexandermalyga
- [ ] [Cloud ML Engine operators assets (AIP-47) (#26836)](https://github.com/apache/airflow/pull/26836): @bkossakowska
## Provider [grpc: 3.1.0rc1](https://pypi.org/project/apache-airflow-providers-grpc/3.1.0rc1)
- [ ] [Look for `extra__` instead of `extra_` in `get_field` (#27489)](https://github.com/apache/airflow/pull/27489): @dstandish
- [ ] [Allow and prefer non-prefixed extra fields for GrpcHook (#27045)](https://github.com/apache/airflow/pull/27045): @dstandish
## Provider [hashicorp: 3.2.0rc1](https://pypi.org/project/apache-airflow-providers-hashicorp/3.2.0rc1)
- [ ] [Add Airflow specific warning classes (#25799)](https://github.com/apache/airflow/pull/25799): @eladkal
## Provider [jdbc: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-jdbc/3.3.0rc1)
- [ ] [Allow and prefer non-prefixed extra fields for JdbcHook (#27044)](https://github.com/apache/airflow/pull/27044): @dstandish
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Look for `extra__` instead of `extra_` in `get_field` (#27489)](https://github.com/apache/airflow/pull/27489): @dstandish
## Provider [microsoft.azure: 5.0.0rc1](https://pypi.org/project/apache-airflow-providers-microsoft-azure/5.0.0rc1)
- [ ] [Remove deprecated classes in Azure provider (#27417)](https://github.com/apache/airflow/pull/27417): @eladkal
- [ ] [Add azure, google, authentication library limits to eager upgrade (#27535)](https://github.com/apache/airflow/pull/27535): @potiuk
- [ ] [Allow and prefer non-prefixed extra fields for remaining azure (#27220)](https://github.com/apache/airflow/pull/27220): @dstandish
- [ ] [Allow and prefer non-prefixed extra fields for AzureFileShareHook (#27041)](https://github.com/apache/airflow/pull/27041): @dstandish
- [ ] [Allow and prefer non-prefixed extra fields for AzureDataExplorerHook (#27219)](https://github.com/apache/airflow/pull/27219): @dstandish
- [ ] [Allow and prefer non-prefixed extra fields for AzureDataFactoryHook (#27047)](https://github.com/apache/airflow/pull/27047): @dstandish
- [ ] [Update WasbHook to reflect preference for unprefixed extra (#27024)](https://github.com/apache/airflow/pull/27024): @dstandish
- [ ] [Look for `extra__` instead of `extra_` in `get_field` (#27489)](https://github.com/apache/airflow/pull/27489): @dstandish
- [ ] [Fix Azure Batch errors revealed by added typing to azure batch lib (#27601)](https://github.com/apache/airflow/pull/27601): @potiuk
- [ ] [Fix separator getting added to variables_prefix when empty (#26749)](https://github.com/apache/airflow/pull/26749): @rajaths010494
## Provider [microsoft.mssql: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-microsoft-mssql/3.3.0rc1)
- [ ] [Remove unnecessary newlines around single arg in signature (#27525)](https://github.com/apache/airflow/pull/27525): @dstandish
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [microsoft.winrm: 3.1.0rc1](https://pypi.org/project/apache-airflow-providers-microsoft-winrm/3.1.0rc1)
- [ ] [A few docs fixups (#26788)](https://github.com/apache/airflow/pull/26788): @blag
## Provider [mongo: 3.1.0rc1](https://pypi.org/project/apache-airflow-providers-mongo/3.1.0rc1)
- [ ] [Fix links to sources for examples (#24386)](https://github.com/apache/airflow/pull/24386): @potiuk
## Provider [mysql: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-mysql/3.3.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [opsgenie: 5.0.0rc1](https://pypi.org/project/apache-airflow-providers-opsgenie/5.0.0rc1)
- [ ] [Remove deprecated code from Opsgenie provider (#27252)](https://github.com/apache/airflow/pull/27252): @eladkal
## Provider [oracle: 3.5.0rc1](https://pypi.org/project/apache-airflow-providers-oracle/3.5.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [postgres: 5.3.0rc1](https://pypi.org/project/apache-airflow-providers-postgres/5.3.0rc1)
- [ ] [PostgresHook: Added ON CONFLICT DO NOTHING statement when all target fields are primary keys (#26661)](https://github.com/apache/airflow/pull/26661): @alexandermalyga
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Rename schema to database in PostgresHook (#26744)](https://github.com/apache/airflow/pull/26744): @feluelle
## Provider [salesforce: 5.2.0rc1](https://pypi.org/project/apache-airflow-providers-salesforce/5.2.0rc1)
- [ ] [Allow and prefer non-prefixed extra fields for SalesforceHook (#27075)](https://github.com/apache/airflow/pull/27075): @dstandish
## Provider [sendgrid: 3.1.0rc1](https://pypi.org/project/apache-airflow-providers-sendgrid/3.1.0rc1)
- [ ] [Add Airflow specific warning classes (#25799)](https://github.com/apache/airflow/pull/25799): @eladkal
## Provider [sftp: 4.2.0rc1](https://pypi.org/project/apache-airflow-providers-sftp/4.2.0rc1)
- [ ] [SFTP Provider: Fix default folder permissions (#26593)](https://github.com/apache/airflow/pull/26593): @MatthieuBlais
## Provider [slack: 7.0.0rc1](https://pypi.org/project/apache-airflow-providers-slack/7.0.0rc1)
- [ ] [Allow and prefer non-prefixed extra fields for slack hooks (#27070)](https://github.com/apache/airflow/pull/27070): @dstandish
## Provider [snowflake: 4.0.0rc1](https://pypi.org/project/apache-airflow-providers-snowflake/4.0.0rc1)
- [ ] [Update snowflake hook to not use extra prefix (#26764)](https://github.com/apache/airflow/pull/26764): @dstandish
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
- [ ] [Use unused SQLCheckOperator.parameters in SQLCheckOperator.execute. (#27599)](https://github.com/apache/airflow/pull/27599): @wjmolina
## Provider [sqlite: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-sqlite/3.3.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [ssh: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-ssh/3.3.0rc1)
- [ ] [Added docs regarding templated field (#27301)](https://github.com/apache/airflow/pull/27301): @bdsoha
- [ ] [Added environment to templated SSHOperator fields (#26824)](https://github.com/apache/airflow/pull/26824): @bdsoha
- [ ] [Apply log formatter on every ouput line in SSHOperator (#27442)](https://github.com/apache/airflow/pull/27442): @dolfinus
- [ ] [A few docs fixups (#26788)](https://github.com/apache/airflow/pull/26788): @blag
- [ ] [SSHOperator ignores cmd_timeout (#27182) (#27184)](https://github.com/apache/airflow/pull/27184): @punx120
## Provider [tableau: 4.0.0rc1](https://pypi.org/project/apache-airflow-providers-tableau/4.0.0rc1)
- [ ] [Remove deprecated Tableau classes (#27288)](https://github.com/apache/airflow/pull/27288): @eladkal
## Provider [trino: 4.2.0rc1](https://pypi.org/project/apache-airflow-providers-trino/4.2.0rc1)
- [ ] [Bump Trino version to fix non-working DML queries (#27168)](https://github.com/apache/airflow/pull/27168): @alexandermalyga
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [vertica: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-vertica/3.3.0rc1)
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [yandex: 3.2.0rc1](https://pypi.org/project/apache-airflow-providers-yandex/3.2.0rc1)
- [ ] [Allow no extra prefix in yandex hook (#27040)](https://github.com/apache/airflow/pull/27040): @dstandish
## Provider [zendesk: 4.1.0rc1](https://pypi.org/project/apache-airflow-providers-zendesk/4.1.0rc1)
- [ ] [fix zendesk change log (#27363)](https://github.com/apache/airflow/pull/27363): @eladkal
The guidelines on how to test providers can be found in
[Verify providers by contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors)
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bdsoha commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
bdsoha commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1318996008
Tested #27404, #27376, #27321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317645508
> This seems a little weird to me because the fix for the issue was to bump the dependency to trino>=0.318.0 in the first place, so is there any point in PRs that just bump some dependency version, if these versions are getting downloaded into old releases anyway?
Of course - it's the installation of new provider bumped trino upgrade. If you already had an earlier version of trino library installed the only way to upgrade it would be to manually upgrade the trino library. If we did not have >= 0.318 it would not have happened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320993162
Hello @alexott - I have always been valued the work of yours. And please don't treat that perspnally. This comment has no personal attack on you in any way, but your comment come to me as a bit of shock. And I presume you are speaking in the name of Databricks as an organisation here. If not, I am even a bit more shocked.
Are you seriously saying that the work of volunteers that prepare and release the ASF software mostly in their free (or sponsored by some responsible players) should be impacted to delay/no time that commercial company - Databricks - interested in their (Databricks) intergration could not afford any of their employees to spend any time on it during their day job?
Do I interpret your words well?
Are you seriously saying that Databricks has an expectation that their integration will Airflow works and get tested, but they are not able to allocate time of their employees working on it to verify it during their day job, and instead they expect that Airflow PMC process should be changed to impact (and delay) general release process of 70+ providers because of that?
If that's the case, I am literally shocked - even stunned. It shows great misunderstanding of the role and contributions that the company can make to the project that they deeply care about. (again - I have nothing to you personally, It's a comment to the organization of yours).
> The problem here was that voting was open only during the work week, not over the weekend as usual. And I have time only over weekend. I agree about having test coverage, but it work in progress.
There is no "usual" thing here. I checked and in the past sometimes voting on Sat, sometimes on Mon, but sometimes on Thu. The 72 hours that the ASF requires is there for a reason. It should give not only a lot of time for reaction, but also should account for geographical locations
https://www.apache.org/foundation/voting.html
> Voting periods should generally run for at least 72 hours to provide an opportunity for all concerned persons to participate, regardless of their geographic location.
Yes. There is "at least", but there is nothig about "make sure it goes through" weekend. Surely companies like Databricks have on-call-rotation with "hours" SLAs for their critical problems, and giving 72 Hrs notice is quite a lot of time to be able to react for the company.
Do I understand the situation correctly?
Just another comment there - If your organisation wants to really have a control over the release process and testing - it's perfectly possible. We could actually let databricks release their own provider (not as a community one). Databricks is free to do that - we can move the "apache-airflow-providers-databricks" and let databricks release their own "databricks-airflow-provider" (same as Great Expectations did). Then you will have fulll control over release, testing and changes. You (Databricks) can do it very easily.
Being part of the community and ASF-ruled project is that you take both benefits and limitations that are coming from that. We do all the release process, we have a number of people (like @kazanzhy ) who contribute and improve the stuff there. This all comes as a benefit for Databricks, as they don't have to necessary put their own effort there into improving and maintaining the libraries. And it comes with the cost - the cost is that release process of ASF has to be followed. The cost is that if you should make sure the code is sufficiently covered by unit tests. Finally the cost is that when we put it up for release Databricks have 72 hrs to see if they are ok with the release - with all the prior notifications, warnings and even listing details list of changes so that Databricks can focus just on this.
And even more - with the AIP-47, you have an apportunity (as Databricks) to create, maintain, develop and RUN system tests for your provider. We have a very well defined way how to build and add those tests, yet we EXPECT Databricks ( similarly as Amazon and Google who pave the way) to eventually run and report the status of their end-2-end integration of the provider. That would have caught the problem way earlier. And you are absolutely free to invest your time and effort to make those examples/system_tests fully automatically runnable, have an account to run them and actually run and report result of those on whatever frequency you think is appropriate. This is something Databricks could do today. Just needs to invest. And this is the way how the "testability" of the Databricks provider can be improved. Not by shifting voting to weekends.
I'd really love that Databricks does the investment there to improve the test quality tehre. If they care about the quality of their integration - AIP-47 is precisely about that. And the best way of doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317008681
> We are getting the below error when trying to install the Kubernetes RC via setup.cfg:
My guess you might have the other RC`s already installed locally and explicitly that's why they are found by `pip` - but if you do it through extra and they are not installed manually, `pip` does not find pre-release candidates automatically - you need to tell it to search also among pre-release candidates by `--pre` switch of `pip install`.
```
--pre Include pre-release and development versions. By default, pip only finds stable versions.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320910427
@potiuk unfortunately DatabricksSqlOperator is broken - it fails with error:
```
AttributeError: 'schema' is configured as a template field but DatabricksSqlOperator does not have this attribute.
```
most probably it's caused by #25717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1319774555
Unfortunately I'll have time to test Databricks provider only on weekend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320928386
We are always looking for ways to improve :)
If you specific idea in mind about what we can do better (procedure wise) I suggest to start a thread in the mailing list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320911452
Can we revert this release of databricks provider until this issue is investigated & fixed? It's really bad that refactoring was done without testing of affected operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1314609894
Note that `cncf.kubernetes is rc3 - because I accidentally published cncf.kubernetes 5.0.0rc1 and rc2 some time in the past and it is impossible to delet an already released package in PyPI so I had to use rc3 for the `cncdf.kubernetes` provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hankehly commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
hankehly commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320994936
The following work as expected (better late than never)
- https://github.com/apache/airflow/pull/27076
- https://github.com/apache/airflow/pull/27564
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320998604
(but this is not something I feel strongly about. Just my own view)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rajaths010494 commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
rajaths010494 commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1316413953
@potiuk The following work great:
Provider [microsoft.azure: 5.0.0rc1](https://pypi.org/project/apache-airflow-providers-microsoft-azure/5.0.0rc1)
- [x] https://github.com/apache/airflow/pull/26749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317853138
ok i'll keep compiling things in that PR and yeah, somethings can be deferred for sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320996111
> I dont think there is need to yank the release because one operator stopped working. Affected users can use older version of provider while a fix is in progress till next release.
I think yanking is fine for that -> this is really a "soft" way of making releases not "searchable" when doing upgrades. People still can choose to install them manually, no problem. And sinece we have not included the new providers in "2.4.3" - it has no impact for the default bugfix installations/image/constraints versions.
I personally believe we should use yanking more frequently (and have no objections for it if there is a good reason). for releases we do not want to install "a new" - because we know they are somewhat seriously flawed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1321773589
@potiuk sorry for adding more work on your shoulders, and I appreciate your work & other maintainers. We still don't have on-call for Airflow operators. Work on improving test coverage, and setting up continuous testing is prioritized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Voldurk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
Voldurk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1324133151
@potiuk Please excuse the delay, #27033 tested OK on Composer+Airflow 2.3.4, but had to resolve dependency issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexandermalyga commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexandermalyga commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317595566
@potiuk
Provider [trino: 4.2.0rc1](https://pypi.org/project/apache-airflow-providers-trino/4.2.0rc1) https://github.com/apache/airflow/pull/27168
**This particular issue is fixed in `4.2.0rc1`.**
But, when I tried with `4.1.0` to check that the issue was still in the old version, it wasn't anymore. I guess this is because the dependency was specified as `trino>=0.301.0` in `4.1.0` and it dowloaded `0.319.0` when I installed (which fixes the underlying issue).
This seems a little weird to me because the fix for the issue was to bump the dependency to `trino>=0.318.0` in the first place, so is there any point in PRs that just bump some dependency version, if these versions are getting downloaded into old releases anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317864992
ok @potiuk, in addition to yandex, **jdbc** references a non-existent release in the changelog
but the other suggestions i think can be docs-only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1316612284
#26784 - Tested on actual both AWS SSM Parameter Store and AWS Secrets Manager and circular error gone (I hope forever)
#26687 - Tested non-doc parts
- Test connection show expected error in the UI (non-testable connection)
- Show warning if user provide non-existed `emr_conn_id`
#26946 - Assume role with credentials fixed and work as initial expected
#27134 - Most of changes related to documentation but anyway test that obtain connection in case if it stored in AWS SSM as JSON
#26953 - Works as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1319842627
> Unfortunately I'll have time to test Databricks provider only on weekend
We can always fix in upcoming release if we find problems @alexott :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317860425
ok i completed review and all suggestions are in that PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320916051
The problem here was that voting was open only during the work week, not over the weekend as usual. And I have time only over weekend. I agree about having test coverage, but it work in progress.
P.S. It looks like that #23971 broke saving results to file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bdsoha commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
bdsoha commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1316398126
@potiuk The following work great:
### Provider [ssh: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-ssh/3.3.0rc1)
- [x] https://github.com/apache/airflow/pull/27301
- [x] https://github.com/apache/airflow/pull/26824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
pankajkoti commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1316417611
We are getting the below error when trying to install the Kubernetes RC via `setup.cfg`:
```
ERROR: Could not find a version that satisfies the requirement apache-airflow-providers-cncf-kubernetes==5.0.0rc3; extra == "all" (from astronomer-providers[all]) (from versions: 2.0.2, 3.0.0, 4.0.0, 4.1.0, 4.3.0, 4.4.0)
ERROR: No matching distribution found for apache-airflow-providers-cncf-kubernetes==5.0.0rc3; extra == "all"
```
Any hint that could help us solve this problem? The same via pip install works. But somehow it is failing with setup.cfg only for this RC, all other RCs work fine via `setup.cfg` including `apache-airflow-providers-amazon==6.1.0rc1, apache-airflow-providers-apache-hive==4.1.0rc1, apache-airflow-providers-apache-livy==3.2.0rc1, apache-airflow-providers-common-sql==1.3.0rc1, apache-airflow-providers-databricks==3.4.0rc1, apache-airflow-providers-google==8.5.0rc1, apache-airflow-providers-microsoft-azure==5.0.0rc1, apache-airflow-providers-snowflake==4.0.0rc1, apache-airflow-providers-trino==4.2.0rc1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
pankajkoti commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317062700
> > We are getting the below error when trying to install the Kubernetes RC via setup.cfg:
>
> My guess you might have the other RCs already installed locally and explicitly that's why they are found by `pip` - but if you do it through extra and they are not installed manually, `pip` does not find pre-release candidates automatically - you need to tell it to search also among pre-release candidates by `--pre` switch of `pip install`.
>
> ```
> --pre Include pre-release and development versions. By default, pip only finds stable versions.
> ```
Thanks Jarek. I am sorry for the bother here, looks like it is an astronomer internal issue we have because of a constraint we have on the kubernetes provider releases. @kaxil helped us figure this out, so thanks to Kaxil too :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
pankajastro commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1318984985
Tested #26652, #27330, #27533, #27236, #27543 work as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1319770027
Thanks everyone. Closing this one and releasing most providers (see announcement at devlist for details)l
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] punx120 commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
punx120 commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1318756491
#27184 Tested - looks good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1323396660
@alexott - Also to explain more (and apologise, if it's been a little - or even little more than a little - harsh).
But I think this is really just question of balance between the risk we will break something and the time/effort of somone's (who?) to do enough testiing to be confident envouhg. There is never 100%. We cannot extend the testing period for very long time because it will hamper our "operations". And we do not know who is / is not available for testing. We cannot really have a point to say "we wait for response".
The most we can do is to make all the reasonable effort to flag and notify everyone involved and give them SOME time to react. Is 72 hours enough? Should it be 96 maybe? Should we always have weekend included?
To be honest I was quite surprised to see the statement of yours - usually we made voting last 1 more day if the weekend was included to account that people will NOT have time to do testing then because usually they spend weekends with their families etc. I often (my personal choice) do some CI stuff during wekends because then it's faster and safer to iterate on it) and quite often I shift my working/non-working days freely because I have very flexible situation (no day job, no kids, wife that has also super flexible schedule), but I would not think this is "needed" by others :)
Do you really think having a weekend as part of the voting time for providers is a good idea and way to improve the quality of testing in general? I think we COULD make sure we always start voting on Wed and finish on Tuesday, I just have feeling that would be rather artifficial setting for one specific case only. I often did adjust the relase schedule to various events (for example I would never start voting this Wednesday - knowing that ThanksGiving is coming),.
Maybe this is also question to others in this thread. We have quite representative group in this thread - do you think that would be a good idea to add some regularity in the day-of-week process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexandermalyga commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexandermalyga commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317657079
[Make GSheetsHook return an empty list when there are no values (#27261)](https://github.com/apache/airflow/pull/27261) works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexandermalyga commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
alexandermalyga commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317692819
[PostgresHook: Added ON CONFLICT DO NOTHING statement when all target fields are primary keys #26661](https://github.com/apache/airflow/pull/26661) works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] r-richmond commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
r-richmond commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1315564598
Attempted to test `google: 8.5.0rc1` by installing it into a dev instance and was hit with the following error. Which I suppose is expected considering `apache-airflow-providers-common-sql 1.3.0` has rc1 above but I don't have an easy way to install `8.5.0rc1` in my dev environment as a result.
One workaround would be to update `google 8.5.0rcX` to be dependent on `apache-airflow-providers-common-sql>=1.3.0rX`
```
Because apache-airflow-providers-google (8.5.0rc1) depends on apache-airflow-providers-common-sql (>=1.3.0) which doesn't match any versions, apache-airflow-providers-google is forbidden.
So, because airflow-dev depends on apache-airflow-providers-google (==8.5.0rc1), version solving failed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1316243022
@r-richmond I cannot reproduce it. Where did you install the rc1 package from ?
I just double checked and our RC packages have mechanism implemented to cope with the issue you mentioned and they properly depend on .* version in case of such a cross-dependency:
```
Requires-Dist: apache-airflow-providers-common-sql (>=1.3.0.*)
```
The `.*` is added automatically when we release and RC package if we depend on another `apache-airflow` package - specificaly to handle such case.
When I try to install `apache-airflow-providers-google==8.5.0rc1`, it properly installs and pulls common-sql==1.3.0rc1
```
...
Installing collected packages: chardet, asgiref, apache-airflow-providers-common-sql, aiofiles, gcloud-aio-auth, gcloud-aio-storage, gcloud-aio-bigquery, google-cloud-dataform, google-cloud-compute, apache-airflow-providers-google
Attempting uninstall: apache-airflow-providers-common-sql
Found existing installation: apache-airflow-providers-common-sql 1.1.0
Uninstalling apache-airflow-providers-common-sql-1.1.0:
Successfully uninstalled apache-airflow-providers-common-sql-1.1.0
Attempting uninstall: apache-airflow-providers-google
Found existing installation: apache-airflow-providers-google 8.2.0
Uninstalling apache-airflow-providers-google-8.2.0:
Successfully uninstalled apache-airflow-providers-google-8.2.0
Successfully installed aiofiles-0.8.0 apache-airflow-providers-common-sql-1.3.0rc1 apache-airflow-providers-google-8.5.0rc1 asgiref-3.5.2 chardet-4.0.0 gcloud-aio-auth-4.0.1 gcloud-aio-bigquery-6.1.2 gcloud-aio-storage-7.0.1 google-cloud-compute-0.7.0 google-cloud-dataform-0.2.0
```
So I wonder where your errors came from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320913416
> It's really bad that refactoring was done without testing of affected operators.
We always welcome the community to test. This is why we have this tracker before release.
Noting that if provider is not well covered with unit tests then we have no way of knowing if new functionality works. Other providers have coverage for testing templated fields.
I dont think there is need to yank the release because one operator stopped working. Affected users can use older version of provider while a fix is in progress till next release.
Fixing the template field issue should be simple enough cc @kazanzhy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320997378
> @eladkal - I think yanking is fine for that -> this is really a "soft" way of making releases not "searchable" when doing upgrades. People still can choose to install them manually, no problem. And sinece we have not included the new providers in "2.4.3" - it has no impact for the default bugfix installations/image/constraints versions.
>
> I personally believe we should use yanking more frequently (and have no objections for it if there is a good reason) - for releases we do not want tolet users install "a new" - because we know they are somewhat seriously flawed.
I'm not sure I agree.
Yanking is harsh. It means under no circumstances do not use this version. This is not the case here. 1 functionality out of many others has regression. If someone use this package without this specific operator why should he be effected? More over, if that individual waited for another feature you are now preventing him from using this new feature.
To me this sounds like this path means always yank everything because almost any new release fix regressions from previous ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1315676536
Actually, the RC versions shouls also support dependencies to other rc packages. but I will check it later (on the run now). You can workaround and install both packages with `--no-deps` switch fi you have problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1316261271
OK. I think I know. From the error message, seems you are using poetry @r-richmond. Poetry is not supported to install airflow, and it has apparently a bug that it cannot properly handle the vaild >= .* requirement. You MUST use `pip` to install airflow I am afraid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317794272
@potiuk the yandex changelog has a note about the 4.0 release since i thought that we would be bumping.
see here https://github.com/apache/airflow/pull/27040/files#diff-78feedeed4b63a7326f7ade03a383d397cceb5f48c2eda0f90ddcce25494033aR27
confirmed it's in the release package. i am going through all of them now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1317826050
> yandex release notes update: release is 3.2 not 4.0
Ok. Thanks. Missed it. I will remove yandex and make a follow-up release.
> asana can be minor
I will leave as it is. It was originally marked as breaking, and maybe even if it is not, it's still OK from SemVer point of view (we are ok to bump major version any time we want, regardless. This is at most small annoyance.
> k8s prefix removal is feature not bug
Will leave it as is, This can be corrected in the docs (in the next version), it will stay in the PIP/Package README though. But again, it's minor inconvenience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sshivprasad commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
sshivprasad commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1319789306
Tested #26951 - Works as expected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27674: Status of testing Providers that were prepared on November 15, 2022
URL: https://github.com/apache/airflow/issues/27674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] emincanoguz11 commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
emincanoguz11 commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1319831061
Hi @potiuk. Sorry for late response, I am very busy nowadays.
I tested #27174 - Its works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1320994794
And yes - it is the weekemd, where I am on the woods and I just yanked the release (it's reversible as well):
<img width="939" alt="Screenshot 2022-11-20 at 00 54 08" src="https://user-images.githubusercontent.com/595491/202876165-f10b2f08-2d9a-4223-a28a-c1b177ca4bff.png">
I am planning the follow-up release right after I am back (Monday afternoon) so if we will find and investigate/fix the issue before that - I am happy to include 3.4.1 databricks provider in this wave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27674: Status of testing Providers that were prepared on November 15, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27674:
URL: https://github.com/apache/airflow/issues/27674#issuecomment-1323420644
@eladkal I looked at the change again, and I think I want to keep my approch - yanking - especially if we can fix it quickly - @alexott @kazanzhy ? Can we have a fix for the problem today/tomorrow? I would then release a fixed version.
Why I think you should yank it ?
SQL interface is an extremely important part of any modern data interface. Even if this is **just** SQLExecute, people will rely on it heavily and by upgrading to the new databricks release a lot of the workflows might be broken. Databricks is popular and important part of our ecosystem. With this bug, we will not only impact Databricks "view" but also Airflow's one and we might have to deal with many issues raised by our users. Yanking is the way to prevent those issues from happening in most cases. But it also does not prevent those who want to use new features released in the new Databricks Provider. They STILL can install the provider if they specify `==3.4.0` . No problem with that. This is precisely what yanking provides - it means that you won't accidentally install it when you upgrade by running `pip install --upgrade` or when installing Airlfow from the scratch without constraints. But you still can install this version if you need the features and you do not care about
SQL error. You will see the message why installing it (including the reason for yanking that currently explains that there is a bug in SQL operator). I think this is the best approach for now.
Especially if we will be able to release a fix in few days, this will be only a minor inconvenience that users won't be able to use the new features of Databricks provider "automatically". Compare that with potential "radius blast" of broken SQL operator for Databricks. It's like few orders of magnitude more serious problem that our users might have to deal with if we do not yank it now.
So @kazanzhy @alexott - is it possible to fix this problem quickly ? If so - I am happy to wait with the RC2 candidate release I am going to do for those providers that we skipped in last wave until this one is tested and merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org