[GitHub] [airflow] ldacey commented on pull request #6850: [AIRFLOW-6296] add ODBC hook & deprecation warning for pymssql

[GitBox <gi...@apache.org>]

ldacey commented on pull request #6850:
URL: https://github.com/apache/airflow/pull/6850#issuecomment-831989010


   Recently added support for Snowflake so I thought I would share it in case anyone else is using turbodbc and Airflow. The nice part about this is that my data goes straight from the database to a partitioned parquet dataset without worrying about types changing (due to pandas or reading in CSV data). I can use a single "OdbcToAzureBlobOperator" for all of my SQL data sources.
   
   Driver installation in my Dockerfile:
   
   ```
     && curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
       && apt-get update && apt-get -y dist-upgrade \
       && curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list \
       && apt-get update \
       && ACCEPT_EULA=Y apt-get install -y \
           msodbcsql17 \
           mssql-tools \
       && curl -SL https://dev.mysql.com/get/downloads/mysql-connector-odbc-8.0.23-linux-glibc2.12-x86-64bit.tar.gz | tar -zxC /opt \
       && cp /opt/mysql-connector-odbc-8.0.23-linux-glibc2.12-x86-64bit/lib/libmyodbc8* /usr/lib/x86_64-linux-gnu/odbc/ \
       && /opt/mysql-connector-odbc-8.0.23-linux-glibc2.12-x86-64bit/bin/myodbc-installer -d -a -n "MySQL ODBC 8.0 ANSI Driver" -t "DRIVER=/usr/lib/x86_64-linux-gnu/odbc/libmyodbc8a.so;" \
       && /opt/mysql-connector-odbc-8.0.23-linux-glibc2.12-x86-64bit/bin/myodbc-installer -d -a -n "MySQL ODBC 8.0 Unicode Driver" -t "DRIVER=/usr/lib/x86_64-linux-gnu/odbc/libmyodbc8w.so;" \
       && curl -SL https://sfc-repo.azure.snowflakecomputing.com/odbc/linux/2.23.1/snowflake_linux_x8664_odbc-2.23.1.tgz | tar -zxC /opt \
       && mv /opt/snowflake_odbc /usr/lib/x86_64-linux-gnu/odbc/ \
       && /usr/lib/x86_64-linux-gnu/odbc/snowflake_odbc/iodbc_setup.sh \
   ```
   
   Sample of Extras from each database type:
   
   Snowflake:
   ```
   {"Driver": "SnowflakeDSIIDriver", "warehouse": "warehousename", "role": "rolename"}
   ```
   
   MS SQL:
   ```
   {"Driver": "ODBC Driver 17 for SQL Server", "ApplicationIntent": "ReadOnly", "TrustedConnection": "Yes"}
   ```
   
   MySQL:
   ```
   {"Driver": "MySQL ODBC 8.0 Unicode Driver", "Threading": 2, "INITSTMT": "set session time_zone ='+00:00'"}
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org