You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/02 01:02:04 UTC
[GitHub] [airflow] mariotaddeucci opened a new pull request #19358: Fix S3 to redshift operator
mariotaddeucci opened a new pull request #19358:
URL: https://github.com/apache/airflow/pull/19358
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19358:
URL: https://github.com/apache/airflow/pull/19358#issuecomment-958236613
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mariotaddeucci commented on pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
mariotaddeucci commented on pull request #19358:
URL: https://github.com/apache/airflow/pull/19358#issuecomment-958356517
@john-jac by using UPSERT or REPLACE with new RedshiftSqlHook (changed to remove postgres provider depency), the operator executes multiple queries in a transaction. Different from postgres hook, RedshiftSqlHook don't support multiple commands into a prepared statement, to fix the operator was necessary change the string multi query to a list of queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19358:
URL: https://github.com/apache/airflow/pull/19358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19358:
URL: https://github.com/apache/airflow/pull/19358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] john-jac edited a comment on pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
john-jac edited a comment on pull request #19358:
URL: https://github.com/apache/airflow/pull/19358#issuecomment-958307133
> Bug happens on S3ToRedshiftOperator with specific configuration. By using "UPSERT" or "REPLACE" is generate an sql block with multiple queries. The RedshiftSQLHook don't support execute multiple queries in a single call of execute. To fix it just need to convert the single query string to a list of queries.
Hi @mariotaddeucci . What Redshift configuration causes the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mariotaddeucci commented on pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
mariotaddeucci commented on pull request #19358:
URL: https://github.com/apache/airflow/pull/19358#issuecomment-958356517
@john-jac by using UPSERT or REPLACE with new RedshiftSqlHook (changed to remove postgres provider depency), the operator executes multiple queries in a transaction. Different from postgres hook, RedshiftSqlHook don't support multiple commands into a prepared statement, to fix the operator was necessary change the string multi query to a list of queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19358:
URL: https://github.com/apache/airflow/pull/19358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] john-jac commented on pull request #19358: Fix S3ToRedshiftOperator
Posted by GitBox <gi...@apache.org>.
john-jac commented on pull request #19358:
URL: https://github.com/apache/airflow/pull/19358#issuecomment-958307133
> Bug happens on S3ToRedshiftOperator with specific configuration. By using "UPSERT" or "REPLACE" is generate an sql block with multiple queries. The RedshiftSQLHook don't support execute multiple queries in a single call of execute. To fix it just need to convert the single query string to a list of queries.
Hi @ mariotaddeucci . What Redshift configuration causes the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org