You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/24 15:02:30 UTC
[GitHub] [airflow] potiuk opened a new issue, #27894: Status of testing Providers that were prepared on November 24, 2022
potiuk opened a new issue, #27894:
URL: https://github.com/apache/airflow/issues/27894
### Body
I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?
Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial changes introduced:
## Provider [amazon: 6.2.0rc1](https://pypi.org/project/apache-airflow-providers-amazon/6.2.0rc1)
- [ ] [Use Boto waiters instead of customer _await_status method for RDS Operators (#27410)](https://github.com/apache/airflow/pull/27410): @hankehly
- [ ] [Handle transient state errors in `RedshiftResumeClusterOperator` and `RedshiftPauseClusterOperator` (#27276)](https://github.com/apache/airflow/pull/27276): @syedahsn
- [ ] [Correct job name matching in SagemakerProcessingOperator (#27634)](https://github.com/apache/airflow/pull/27634): @ferruzzi
## Provider [asana: 2.1.0rc1](https://pypi.org/project/apache-airflow-providers-asana/2.1.0rc1)
- [ ] [Allow and prefer non-prefixed extra fields for AsanaHook (#27043)](https://github.com/apache/airflow/pull/27043): @dstandish
## Provider [common.sql: 1.3.1rc1](https://pypi.org/project/apache-airflow-providers-common-sql/1.3.1rc1)
- [ ] [Restore removed (but used) methods in common.sql (#27843)](https://github.com/apache/airflow/pull/27843): @potiuk
- [ ] [Fix errors in Databricks SQL operator introduced when refactoring (#27854)](https://github.com/apache/airflow/pull/27854): @potiuk
## Provider [databricks: 4.0.0rc1](https://pypi.org/project/apache-airflow-providers-databricks/4.0.0rc1)
- [ ] [Fix errors in Databricks SQL operator introduced when refactoring (#27854)](https://github.com/apache/airflow/pull/27854): @potiuk
- [ ] [Fix templating fields and do_xcom_push in DatabricksSQLOperator (#27868)](https://github.com/apache/airflow/pull/27868): @potiuk
## Provider [exasol: 4.1.1rc1](https://pypi.org/project/apache-airflow-providers-exasol/4.1.1rc1)
- [ ] [Fix errors in Databricks SQL operator introduced when refactoring (#27854)](https://github.com/apache/airflow/pull/27854): @potiuk
## Provider [google: 8.6.0rc1](https://pypi.org/project/apache-airflow-providers-google/8.6.0rc1)
- [ ] [Persist DataprocLink for workflow operators regardless of job status (#26986)](https://github.com/apache/airflow/pull/26986): @vksunilk
- [ ] [Deferrable mode for BigQueryToGCSOperator (#27683)](https://github.com/apache/airflow/pull/27683): @lwyszomi
- [ ] [Fix to read location parameter properly in BigQueryToBigQueryOperator (#27661)](https://github.com/apache/airflow/pull/27661): @VladaZakharova
## Provider [jdbc: 3.3.0rc1](https://pypi.org/project/apache-airflow-providers-jdbc/3.3.0rc1)
- [ ] [Allow and prefer non-prefixed extra fields for JdbcHook (#27044)](https://github.com/apache/airflow/pull/27044): @dstandish
- [ ] [Add SQLExecuteQueryOperator (#25717)](https://github.com/apache/airflow/pull/25717): @kazanzhy
## Provider [mysql: 3.4.0rc1](https://pypi.org/project/apache-airflow-providers-mysql/3.4.0rc1)
- [ ] [Allow SSL mode in MySQL provider (#27717)](https://github.com/apache/airflow/pull/27717): @Adityamalik123
## Provider [neo4j: 3.2.1rc1](https://pypi.org/project/apache-airflow-providers-neo4j/3.2.1rc1)
- [ ] [Fix typing problem revealed after recent Neo4J release (#27759)](https://github.com/apache/airflow/pull/27759): @potiuk
## Provider [presto: 4.2.0rc1](https://pypi.org/project/apache-airflow-providers-presto/4.2.0rc1)
- [ ] [Add _serialize_cell method to TrinoHook and PrestoHook (#27724)](https://github.com/apache/airflow/pull/27724): @alexandermalyga
## Provider [slack: 7.1.0rc1](https://pypi.org/project/apache-airflow-providers-slack/7.1.0rc1)
- [ ] [Implements SqlToSlackApiFileOperator (#26374)](https://github.com/apache/airflow/pull/26374): @Taragolis
## Provider [snowflake: 4.0.1rc1](https://pypi.org/project/apache-airflow-providers-snowflake/4.0.1rc1)
- [ ] [Fix errors in Databricks SQL operator introduced when refactoring (#27854)](https://github.com/apache/airflow/pull/27854): @potiuk
## Provider [trino: 4.3.0rc1](https://pypi.org/project/apache-airflow-providers-trino/4.3.0rc1)
- [ ] [Add _serialize_cell method to TrinoHook and PrestoHook (#27724)](https://github.com/apache/airflow/pull/27724): @alexandermalyga
The guidelines on how to test providers can be found in
[Verify providers by contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-by-contributors)
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vksunilk commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
vksunilk commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326959302
#26986 Works as expected. I am able to view the DataprocLink irrespective of Job status
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326710383
Ah I think I see where I made wrong assumption @alexott . looking at it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326587629
Checked my changes.
cc: @alexott @kazanzhy -> would approeciate checking databricks SQL executor integration with the new common-sql provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1327601172
#26374 tested, work as described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326737975
> In Databricks SQL operator (and I believe in others as well), there was following strategy: always return only last result - previous results were always discarded. Primary reason for this was following:
>
> * When you have multiple SQL statements, first one usually create table, inserts, etc. And only when you have select as the last statement, then you get results. This matches the logic of the SQL's `BATCH` statement
> * When you have multiple SQL statements their result may have different schema, but results will be processed only according to the latest schema, not schemas for corresponding result sets
>
> We may need to think a bit about it - should we return results for each of the statements, or not. If yes, then we need to return pairs of description + results for each SQL statement, instead of using only the latest statement
Yes - I noticed that too now. With two caveats:
* depends on the oprator what is the default (no problem)
* it behaves differently when there is an "sql" passed and return_last is true -> then instead of one-element result array it returns the results
It is surprisingly difficult to unwind teh original convoluted behaviour :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326755638
Opened #27897 - tested all file formats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326659796
#27868 works, #27854 unfortunately not - results are double wrapped:
For example for `select * from default.my_***_table, parameters: None` we get: `scalar_results=True`, `results=[Row(id=1, v='test 1'), Row(id=2, v='test 2')]`. And then code:
```
if scalar_results:
list_results: list[Any] = [results]
else:
list_results = results
```
wraps this list into another list. I think that logic should be changed a bit - right now we're collecting results for all SQL statements into a single list although they could have different schemas.
Let me debug it and open another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1327714877
I will have to cancel the vote due to a bug found still in common.sql. Much more complete and comprehensive (with many tests added) fix is coming https://github.com/apache/airflow/pull/27912
I will carry-over the status of testing for the issues that have been already tested (thanks to everyone who helped with testing! ) so hopefully the last round of testing can focus on sql only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27894: Status of testing Providers that were prepared on November 24, 2022
URL: https://github.com/apache/airflow/issues/27894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexott commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
alexott commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326727196
In Databricks SQL operator (and I believe in others as well), there was following strategy: always return only last result - previous results were always discarded. Primary reason for this was following:
* When you have multiple SQL statements, first one usually create table, inserts, etc. And only when you have select as the last statement, then you get results. This matches the logic of the SQL's `BATCH` statement
* When you have multiple SQL statements their result may have different schema, but results will be processed only according to the latest schema, not schemas for corresponding result sets
We may need to think a bit about it - should we return results for each of the statements, or not. If yes, then we need to return pairs of description + results for each SQL statement, instead of using only the latest statement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326699948
> wraps this list into another list. I think that logic should be changed a bit - right now we're collecting results for all SQL statements into a single list although they could have different schemas.
Yeah. that part is a bit not clear about the intentions (or maybe I misunderstood it). Woudl be great if you have a PR indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] alexandermalyga commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
alexandermalyga commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326589981
#27724 works as expected! Finally Trino inserts are 100% working
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] syedahsn commented on issue #27894: Status of testing Providers that were prepared on November 24, 2022
Posted by GitBox <gi...@apache.org>.
syedahsn commented on issue #27894:
URL: https://github.com/apache/airflow/issues/27894#issuecomment-1326737941
#27276 works as expected. System tests using those operators are all passing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org