You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/13 20:52:33 UTC

[GitHub] [airflow] flolas opened a new issue #16979: Support run multiple sql statements from file in JdbcOperator

flolas opened a new issue #16979:
URL: https://github.com/apache/airflow/issues/16979


   **Description**
   
   Support run multiple sql statements from file in JdbcOperator, when the provided file has multiple statements (we can check it with https://github.com/andialbrecht/sqlparse)
   
   **Use case / motivation**
   
   Running single sql file with multiple sentences with JdbcOperator (uses run method from DbApiHook, maybe we can implement here or in JdbcOperator)
   
   **Are you willing to submit a PR?**
   yes
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #16979: Support run multiple sql statements from file in JdbcOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-879618194


   One comment - this should be done on DBAPI hook, not in JdbcOperator. This way all DB operators will be able to use it. Assigning you to it :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-882830072


   Yeah. Sounds like the third time is the charm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-886607126


   Hey @flolas - how is it going ? I saw a few people interested in it :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] grahovam commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
grahovam commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-879861311


   So the idea here is to not break the statements (and execute one by one) inside `TrinoHook` nor `SnowflakeHook` anymore and therefore break the statements (and execute one by one) inside `DBApiHook`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-882830072


   Yeah. Sounds like the third time is the charm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] flolas commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
flolas commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-886722228


   @potiuk Hi! I will work this week on the PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] grahovam commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
grahovam commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-879740214


   I am already using the PostgresOperator and MySqlHook to execute SQL queries that contain multiple queries seperated by semicolons. And both the operator and the hook use DbApiHook for doing so. Am I missing something here? Thank you for any clarification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-882830072


   Yeah. Sounds like the third time is the charm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-879618194


   One comment - this should be definitely done on DBAPI hook, not in JdbcOperator. This way all DB operators will be able to use it. Assigning you to it :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] eladkal commented on issue #16979: Support run multiple sql statements from file in DBApiHook

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #16979:
URL: https://github.com/apache/airflow/issues/16979#issuecomment-879810198


   > I am already using the PostgresOperator and MySqlHook to execute SQL queries that contain multiple queries seperated by semicolons. And both the operator and the hook use DbApiHook for doing so. Am I missing something here? Thank you for any clarification.
   
   Assuming you have a SQL script:
   `Select 1; Select 2;` Both Postgres and MySQL are able to execute this script. However you will not be able to execute this script in Snowflake, Presto nor Trino. This is why for these databases we first parse the script and break the statements one by one:
   
   https://github.com/apache/airflow/blob/6ab00bfcfdfe1213928281527f32ca31b77cce1b/airflow/providers/trino/hooks/trino.py#L111-L112
   
   https://github.com/apache/airflow/blob/6ab00bfcfdfe1213928281527f32ca31b77cce1b/airflow/providers/snowflake/hooks/snowflake.py#L276-L277
   
   After that we can execute the statements one by one.
   
   
   I think though there is a catch here. Executing `Update table a ... ; Update table b...;` may not the same as executing `Update table a ... ;`  and then executing `Update table b...;` when we break the statements there is a question of when the `commit(s)` gets executed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org