You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/16 23:10:18 UTC
[GitHub] [airflow] prinkov opened a new issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
prinkov opened a new issue #9339:
URL: https://github.com/apache/airflow/issues/9339
When I try to get engine from method get_sqlalchemy_engine of class MsSqlHook, I get the engine without specifying a driver.
Therefore, when I try to execute queries from this engine, I get an error, because method returns an engine with the default pyodbc driver, which may not be installed or not configured.
Maybe should return the engine with the pymssql driver, as it is used in the hook MsSqlHook itself and installed with Airflow extensions. Or give the choice of driver for engines
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prinkov closed issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
prinkov closed issue #9339:
URL: https://github.com/apache/airflow/issues/9339
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prinkov commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
prinkov commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645638008
> I thought pyodbc is part of Airflow's requirements? pyodbc should already be installed right?
> https://github.com/apache/airflow/tree/master/requirements
Hmm, if I install Airflow with command (in docker container or host machine):
`pip install apache-airflow[all_dbs,crypto,jdbc,ssh,mysql,mssql]`
and I trying make the queries from the engine obtained from the MsSqlHook, unfortunately I get the error that pyodbc is not installed.
Although MsSqlHook itself working, since it uses pymssql.
and trying to make requests queries from the engine obtained from the hook, unfortunately I get the error that pyodbc is not installed. Although MsSQLHook itself, since it uses pymssql.
Thus, the behavior seems illogical. The hook always uses pymssql, and at the same time always gives pyodbc as the engine. And if the first will work, this does not guarantee that the second also
My use case. I'm trying create table if not exists in ETL operator. And I don't know the database in advance.
def execute(self, context):
engine = self.db_api_hook.get_sqlalchemy_engine()
if not (type(engine.dialect) == MSDialect_pyodbc or engine.dialect.has_table(engine, table)):
create_table(engine)
elif type(engine.dialect) == MSDialect_pyodbc:
query = '''SELECT * FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = '{table}' '''.format(table=table)
tables = self.db_api_hook.get_records(query)
if len(tables) == 0:
self.create_table(engine)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645056572
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] samuelkhtu commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
samuelkhtu commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645702808
Since the mssql hook will be deprecated in Airflow 2.0, maybe you want to give the ODBC hook a try? The pyodbc is part of the import.
airflow/providers/odbc/hooks/odbc.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prinkov commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
prinkov commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645645641
> I thought pyodbc is part of Airflow's requirements? pyodbc should already be installed right?
As I understand it, in SQLAlchemy it is by default.
https://docs.sqlalchemy.org/en/13/core/engines.html#microsoft-sql-server
But there is no in airflow
https://github.com/apache/airflow/blob/6a9c436f6f013e4c548f59db9397e21b32578d85/airflow/providers/microsoft/mssql/hooks/mssql.py#L76
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] samuelkhtu commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
samuelkhtu commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645643634
If I understand correctly, pymssql is going away and no longer supported. That's why the operator is switched to pyodbc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] samuelkhtu commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
samuelkhtu commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645628588
I thought pyodbc is part of Airflow's requirements? pyodbc should already be installed right?
https://github.com/apache/airflow/tree/master/requirements
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prinkov edited a comment on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
prinkov edited a comment on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645645641
> If I understand correctly, pymssql is going away and no longer supported. That's why the operator is switched to pyodbc.
As I understand it, in SQLAlchemy it is by default.
https://docs.sqlalchemy.org/en/13/core/engines.html#microsoft-sql-server
But there is no in airflow
https://github.com/apache/airflow/blob/6a9c436f6f013e4c548f59db9397e21b32578d85/airflow/providers/microsoft/mssql/hooks/mssql.py#L76
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prinkov commented on issue #9339: get_sqlalchemy_engine from MsSqlHook return engine with pyodbc by default
Posted by GitBox <gi...@apache.org>.
prinkov commented on issue #9339:
URL: https://github.com/apache/airflow/issues/9339#issuecomment-645824878
> Since the mssql hook will be deprecated in Airflow 2.0, maybe you want to give the ODBC hook a try? The pyodbc is part of the import.
>
> airflow/providers/odbc/hooks/odbc.py
Thanks, I'll try.
Thanks for your time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org