You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/03/15 21:57:37 UTC
[I] Fix build SQLAlchemy URI in [airflow]
Taragolis opened a new issue, #38195:
URL: https://github.com/apache/airflow/issues/38195
### Body
Original discussion: https://lists.apache.org/thread/8rhmz3qh30hvkondct4sfmgk4vd07mn5
tl;dr; `DbApiHook` assumes that Airflow Connection URI representation is a valid SA URI, but it is not true and never was, it might work in simple cases, however it will fail in some complex one.
https://github.com/apache/airflow/blob/2ab60812a040cb8b760acaf396f625d0d719e4be/airflow/providers/common/sql/hooks/sql.py#L184-L192
This task intend to track fixes into the hooks which based on `DbApiHook`.
Hooks which don't overwrite invalid `DbApiHook.get_uri` implementation
---
- [ ] `MySqlHook` from `mysql` provider. This hook provide support for two different drivers.
+ [ ] `mysqlclient`: https://pypi.org/project/mysqlclient/
+ [ ] `mysql-connector-python`: https://pypi.org/project/mysql-connector-python/
- [ ] OracleHook from `oracle` provider
- [ ] JdbcHook from `jdbc` provider
- [ ] TrinoHook from `trino` provider
- [ ] HiveServer2Hook from `apache.hive` provider
- [ ] ImpalaHook from `apache.impala` provider
- [ ] DatabricksSqlHook from `databricks` provider
- [ ] ExasolHook from `exasol` provider
- [ ] PrestoHook from `presto` provider
- [ ] VerticaHook fromvertica` provider
Hooks which uses `Connection.get_uri` for construct SQLAlchemy URI
---
- [ ] PostgresHook from `postgres` provider
- [ ] MsSqlHook from `microsoft.mssql` provider
- [ ] SqliteHook from `sqlite` provider
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2775143807
Hi, I think I could help with this issue~
Also, I have submitted some PR toward this issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2057482351
> As far as i remember, we recently discuss this case in slack. I guess better introduce `sa_uri` property (or some similar naming) which returns [`sqlalchemy.engine.URL`](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.engine.URL) into the `DbApiHook` without any implementation, e.g. raise `NotImplementedError` and implements it per hook which have implementation for SQAlchemy.
>
> `get_uri` - not clear which URI it is implements, DBApi? SQLAlchemy or even some internal ([ElasticsearchSQLHook](https://github.com/apache/airflow/blob/dec2662190dd4480d0c631da733e19d2ec9a479d/airflow/providers/elasticsearch/hooks/elasticsearch.py#L113))
>
> cc: @potiuk @dstandish
Why not implement another method named get_url and use that one to instantiate the SQLALchemy engine, as the engine can accept an SQLAlchemy URL object. Also the SQLAlchemy URL class has a factory method named create, which easily allows you to create the URL without the hastle of string manipulations, this is how I did it for my custom SQL operator (the one I mentioned on Slack):
```
def get_url(self) -> URL:
return URL.create(
self.scheme,
username=self.connection.login,
password=self.connection.password,
host=self.connection.host,
port=self.connection.port,
)
```
Also see [here ](https://docs.sqlalchemy.org/en/20/core/engines.html#creating-urls-programmatically)the SQL documentation concerning creating URL's programmatically: [https://docs.sqlalchemy.org/en/20/core/engines.html#creating-urls-programmatically](https://docs.sqlalchemy.org/en/20/core/engines.html#creating-urls-programmatically)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "sc250072 (via GitHub)" <gi...@apache.org>.
sc250072 commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3357338794
> > [@sc250072](https://github.com/sc250072) can you or someone else from Teradata handle the TeradataHook task?
>
> [@eladkal](https://github.com/eladkal) let us handle it and let you know
@eladkal https://github.com/apache/airflow/pull/56305 to address this for Teradata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Bowrna (via GitHub)" <gi...@apache.org>.
Bowrna commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2039277534
hello @Taragolis I was checking this issue and seeing if I could fix it.
https://github.com/apache/airflow/blob/0e8f108313d4af0b450581661aeb8ed35e82a8e6/airflow/models/connection.py#L271-L329
Is fixing the get_uri part so that Airflow Connection URI representation becomes a valid SA URI the right way to fix this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Pyasma (via GitHub)" <gi...@apache.org>.
Pyasma commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3306402619
hi i want to work on this issue could you please assign it to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3067222445
@sc250072 can you or someone else from Teradata handle the TeradataHook task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Pyasma (via GitHub)" <gi...@apache.org>.
Pyasma commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3698735579
@eladkal can we close this issue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2058578313
This is how I done it in my custom SQLRowsInsertOperator:
```
@cached_property
def connection(self) -> Connection:
return BaseHook.get_connection(self.conn_id) # type: ignore
@cached_property
def scheme(self) -> str:
return self.connection.extra_dejson["sqlalchemy_scheme"]
@cached_property
def table_name(self) -> str:
return self.get_columns().table_name_with_schema()
@cached_property
def driver(self) -> str:
return self.connection.extra_dejson["driver"]
# TODO: Maybe this should be moved to DbAPiHook
def get_url(self) -> URL:
self.log.info("Connection schema: %s", self.schema)
self.log.info("Connection scheme: %s", self.scheme)
self.log.info("Connection driver: %s", self.driver)
self.log.info("Connection type: %s", self.connection.conn_type)
self.log.info("Connection login: %s", self.connection.login)
self.log.info("Connection password: %s", self.connection.password)
self.log.info("Connection host: %s", self.connection.host)
self.log.info("Connection port: %s", self.connection.port)
return URL.create(
self.scheme,
username=self.connection.login,
password=self.connection.password,
host=self.connection.host,
port=self.connection.port,
)
# TODO: Maybe this method in DbAPiHook should be changed to this instead of calling get_uri() method
def get_sqlalchemy_engine(self, engine_kwargs=None):
if engine_kwargs is None:
engine_kwargs = {}
return create_engine(self.get_url(), **engine_kwargs)
# TODO: Maybe this should be moved to DbAPiHook
@cached_property
def inspector(self):
engine = self.get_sqlalchemy_engine()
self.log.debug("Engine drivername: %s", engine.url.drivername)
self.log.debug("Engine username: %s", engine.url.username)
self.log.debug("Engine password: %s", engine.url.password)
self.log.debug("Engine host: %s", engine.url.host)
self.log.debug("Engine port: %s", engine.url.port)
self.log.debug("Engine database: %s", engine.url.database)
self.log.debug("Engine dialect: %s", engine.dialect.name)
return Inspector.from_engine(engine)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "BrianLii (via GitHub)" <gi...@apache.org>.
BrianLii commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3030309633
Hello, I would like to work on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Pyasma (via GitHub)" <gi...@apache.org>.
Pyasma commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3694594319
Hey i will start working on the last Hook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2042644747
As far as i remember, we recently discuss this case in slack. I guess better introduce `sa_uri` property (or some similar naming) which returns [`sqlalchemy.engine.URL`](https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.engine.URL) into the `DbApiHook` without any implementation, e.g. raise `NotImplementedError` and implements it per hook which have implementation for SQAlchemy.
`get_uri` - not clear which URI it is implements, DBApi? SQLAlchemy or even some internal ([ElasticsearchSQLHook](https://github.com/apache/airflow/blob/dec2662190dd4480d0c631da733e19d2ec9a479d/airflow/providers/elasticsearch/hooks/elasticsearch.py#L113))
cc: @potiuk @dstandish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "BrianLii (via GitHub)" <gi...@apache.org>.
BrianLii commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3243112996
Hi, I'll be working on this issue this week. I had to switch to another task recently, but I'll return to this now. Thank you for the reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3240744169
@BrianLii are you working on the remaining items?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2058303099
> There is new property which intends to returns SQL Alchemy URL, current behaviour of `get_url` is ambiguous and non-all hook return SA URI here, for some of the DB there is no even SA dialects exists, so better keep `get_url` as is and step-by-step migrate to appropriate ones. So if dialect exists and it installed with provider I do not have any objections to add it implementation as `sqlalchemy_url`
Hey Andrey think there is so confusion here. Yes we should definitly leave the get_uri method as it is (not get_url as it doesn’t exist yet) and implement a new get_url method which would then be called by the existing get_sqlalchemy_engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2059307908
> # TODO: Maybe this should be moved to DbAPiHook
> def get_url(self) -> URL:
> self.log.info("Connection schema: %s", self.schema)
> self.log.info("Connection scheme: %s", self.scheme)
> self.log.info("Connection driver: %s", self.driver)
> self.log.info("Connection type: %s", self.connection.conn_type)
> self.log.info("Connection login: %s", self.connection.login)
> self.log.info("Connection password: %s", self.connection.password)
> self.log.info("Connection host: %s", self.connection.host)
> self.log.info("Connection port: %s", self.connection.port)
>
> return URL.create(
> self.scheme,
> username=self.connection.login,
> password=self.connection.password,
> host=self.connection.host,
> port=self.connection.port,
> )
>
I think this is the plan. First, to add `sqlalchemy_url` to all the providers that use it. And, then, to modify to the base class if I am not wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "sc250072 (via GitHub)" <gi...@apache.org>.
sc250072 commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3310652738
> [@sc250072](https://github.com/sc250072) can you or someone else from Teradata handle the TeradataHook task?
@eladkal let me handle it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "BrianLii (via GitHub)" <gi...@apache.org>.
BrianLii commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3067191223
@eladkal I can confirm the open items left on this issue:
- [X] MySqlHook from `mysql` provider.
- [X] OracleHook from `oracle` provider
- [X] JdbcHook from `jdbc` provider
- [X] TrinoHook from `trino` provider
- [ ] HiveServer2Hook from `apache.hive` provider
- [ ] ImpalaHook from `apache.impala` provider
- [ ] **(WIP)** DatabricksSqlHook from `databricks` provider
- [ ] ExasolHook from `exasol` provider
- [ ] PrestoHook from `presto` provider
- [ ] VerticaHook from `vertica` provider
- [ ] TeradataHook from `teradata` provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Pyasma (via GitHub)" <gi...@apache.org>.
Pyasma commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3354552841
Yea I am working on writing tests for presto hooks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Pyasma (via GitHub)" <gi...@apache.org>.
Pyasma commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3698379646
@potiuk can we close this issue now ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3698845667
Yep. Closing it now :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "BrianLii (via GitHub)" <gi...@apache.org>.
BrianLii commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3285744805
Hi, I may not be able to work on this issue in the near term. Could you please unassign me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2045515454
@Taragolis , I am working on this PR - https://github.com/apache/airflow/pull/38871 . This is a separate PR to just update the `DbApiHook` to include a method to return a SqlAlchemy URL that can be reused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2064137257
> Hey Andrey think there is some confusion here. Yes we should definitly leave the get_uri method as it is (not get_url as it doesn’t exist yet) and implement a new get_url method which would then be called by the existing get_sqlalchemy_engine.
Yes, adding new method is the way to go. How to do it, should be proposed and discussed in the PR, we have 20+ sql providers, and I think the proposal should be complete and explain how to deal with cases that can/cannot be mapped. Also the method should explicily - IMHO - call our sqlalachemy, (`getSqlAlchemyURI` ?) and likely allow variations - from what I understand SQLAlchemy can have different URLs not only for different databases, but also for different drivers. Also some of our Connections allows to add different kinds of extras and parameters - not only hosts but other parameters that might be used and passed differently to different drivers.
If we decide to introduce a new "standard" method, in a product like Airflow and common API like DBApiHook, the proposal should provide answers to all the questions that users might ask in order to be mergable. It does not have to handle all of those but at least get enough error handlling and interface specification to explain what is the transition from the Connection form (in all those cases we have DBAPI implementation) -> SQLAlchemy URL. Which variations are handled, how it should work, how to map engine args etc.
I don't think it can be handled by a single implementation, it's more a scaffloiding + implementation for all the implementations and it should be designed nicely with al least a handful (if not all) implementation from the above list to be quite sure that we will be able to handle those cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2057734177
There is new property which intends to returns SQL Alchemy URL, current behaviour of `get_url` is ambiguous and non-all hook return SA URI here, for some of the DB there is no even SA dialects exists, so better keep `get_url` as is and step-by-step migrate to appropriate ones. So if dialect exists and it installed with provider I do not have any objections to add it implementation as `sqlalchemy_url`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2097059514
> Hey Andrey think there is some confusion here. Yes we should definitly leave the get_uri method as it is (not get_url as it doesn’t exist yet) and implement a new get_url method which would then be called by the existing get_sqlalchemy_engine.
@dabla , I think, we are indeed adding `get_url` functionality using `sqlalchemy_url` property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "willianzilli (via GitHub)" <gi...@apache.org>.
willianzilli commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2910845349
this PR [Fix overwrite get_uri for Oracle #48734](https://github.com/apache/airflow/pull/48734) dont work for get_sqlalchemy_conn because oracledb is supported on SqlAlchemy major 2, which cause error `sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:oracle.oracledb`.
Force "+oracledb" in line 465 broke the code
`uri = f"oracle+oracledb://{login}:{password}@{host}:{port}"`
Also, [test_oracle.py](https://github.com/apache/airflow/blob/providers-oracle/4.1.0/providers/oracle/tests/unit/oracle/hooks/test_oracle.py) file dont check get_sqlalchemy_conn method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "BrianLii (via GitHub)" <gi...@apache.org>.
BrianLii commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3119312258
I don’t see a need to re-implement the `get_uri` feature for `HiveServer2`. According to the [Airflow HiveServer2 connection docs](https://airflow.apache.org/docs/apache-airflow-providers-apache-hive/stable/connections/hiveserver2.html), the connection URI format is compatible with the existing `get_uri` in `DbApiHook`. However, I can implement the currently missing `sqlalchemy_url` property to return the `get_uri` value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "guan404ming (via GitHub)" <gi...@apache.org>.
guan404ming commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2872969417
I plan to get back here after `get_df` migration and I think I would handle the remaining later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3353473488
@Pyasma will you work on the remaining items as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-3365993039
Remaining Tasks:
- [ ] HiveServer2Hook from `apache.hive` provider
- [ ] ImpalaHook from `apache.impala` provider
- [ ] PrestoHook from `presto` provider https://github.com/apache/airflow/pull/56298
- [ ] TeradataHook from `teradata` provider https://github.com/apache/airflow/pull/56305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #38195:
URL: https://github.com/apache/airflow/issues/38195#issuecomment-2872898356
@guan404ming what are the open items left on this issue? I see you handled several of them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Build correct SQLAlchemy URI in hooks based on `DbApiHook` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #38195: Build correct SQLAlchemy URI in hooks based on `DbApiHook`
URL: https://github.com/apache/airflow/issues/38195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org