You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eladkal (via GitHub)" <gi...@apache.org> on 2023/04/04 07:32:54 UTC
[GitHub] [airflow] eladkal opened a new issue, #30451: Add defer support for SQLExecuteQueryOperator
eladkal opened a new issue, #30451:
URL: https://github.com/apache/airflow/issues/30451
### Body
The SQLExecuteQueryOperator serves several kind of DBs: Sqlite, PostgresSQL, Snowflake etc..
We want to explore the option to add defer capabilities to this operator which will be very helpful when running long duration queries.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "faridnsh (via GitHub)" <gi...@apache.org>.
faridnsh commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-3864749041
I'm thinking of giving this a try to make a smallest MVP PR with least amount of feature to have defer support with one of the Postgres drivers, maybe picking up from where @dabla left it. I'm generally a pretty confident person who goes head into problems and then realize that it's bigger than I can handle. 😄 I don't have a lot of experience with writing operators for Airflow, so I would appreciate it, if anyone has any suggestions or tips on how I should tackle and implement the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-6123817227
the goal is a SQLExecuteQueryOperator with defer and start and end from trigger :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-2841291404
first layers to make it possible -> https://github.com/apache/airflow/pull/43944 , https://github.com/apache/airflow/pull/49392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-2928791938
I’ve made #49392 ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "developer-rpai (via GitHub)" <gi...@apache.org>.
developer-rpai commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-6134439298
Hi @karenbraganz, nice work on #65618. I'd like to help get it landed rather than open another PR. I can test it against Postgres with long running queries, including trigger timeouts and a triggerer restart mid query. Once Postgres is in, I could take a follow up for Oracle or Snowflake using the same SQLExecuteQueryTrigger, plus a short docs section. Would test notes on the PR help, or is there a piece you'd like someone to take?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "bruno-uy (via GitHub)" <gi...@apache.org>.
bruno-uy commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-1757738643
This would be awesome. We have some long-running queries that would be great if we could defer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-3864795266
> works in progress for SQLExecuteQueryTrigger [#55094](https://github.com/apache/airflow/pull/55094)
I’ve closed this PR as it was an old one and will be fixed in another one as now its easy to fix with common compat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-5239736563
WIP -> https://github.com/apache/airflow/pull/65618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] notatallshaw-gts commented on issue #30451: Add defer support for SQLExecuteQueryOperator
Posted by "notatallshaw-gts (via GitHub)" <gi...@apache.org>.
notatallshaw-gts commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-1496043958
Astronomer provides as Snowflake sensor, you should take a look what is possible with it: https://github.com/astronomer/astronomer-providers#extras
FYI I believe Astronomer have also investigated other databases and it was non-trivial as getting the async to be performant and provide enough information relies on both the Python library and the underlying drivers.
I'm sure PRs would be welcome by Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-3705023919
works in progress for SQLExecuteQueryTrigger https://github.com/apache/airflow/pull/55094/files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add defer support for SQLExecuteQueryOperator [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on issue #30451:
URL: https://github.com/apache/airflow/issues/30451#issuecomment-3864796541
> I'm thinking of giving this a try to make a smallest MVP PR with least amount of feature to have defer support with one of the Postgres drivers, maybe picking up from where [@dabla](https://github.com/dabla) left it. I'm generally a pretty confident person who goes head into problems and then realize that it's bigger than I can handle. 😄 I don't have a lot of experience with writing operators for Airflow, so I would appreciate it, if anyone has any suggestions or tips on how I should tackle and implement the change.
If I was you I would experiment with implementing a PostgresAsyncHook first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org