You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/27 03:18:01 UTC

[GitHub] [airflow] QuinRiva opened a new issue #11874: MSSQLToBigQuery incorrect type conversion bit-->integer

QuinRiva opened a new issue #11874:
URL: https://github.com/apache/airflow/issues/11874


   **Apache Airflow version**: 1.10.12
   
   **Environment**:  Centos 7 Host
   
   - **Cloud provider or hardware configuration**: 
   - **OS** (e.g. from /etc/os-release): Ubuntu 20.04 Docker Container
   - **Kernel** (e.g. `uname -a`): Linux c6c6e8230c17 3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
   - **Install tools**: Docker Compose
   - **Others** ?
   
   **What happened**:
   
   The schema file generated by MSSQLToGCSOperator specifies that a field defined as **bit** in MSSQL, is defined as **INTEGER**.  However, the JSON file defines the value as **true**/**false**.  When attempting to import into BigQuery using the GCSOperatorToBigQuery operator, the job fails:
   `Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value 'boolean_value: false' to integer. Field: manager; Value: 0`
   
   **What you expected to happen**:
   
   Value should be defined as **BOOLEAN** in the schema.
   
   **How to reproduce it**:
   Ensure `dbo.Customers` table has a **bit** column:
   
   ```
   export_customers = MsSqlToGoogleCloudStorageOperator(
                   task_id='export_customers',
                   sql='SELECT * FROM dbo.Customers;',
                   bucket='mssql-export',
                   filename='data/customers/export.json',
                   schema_filename='schemas/export.json',
                   mssql_conn_id='mssql_default',
                   google_cloud_storage_conn_id='google_cloud_default',
                   dag=dag
               )
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #11874: MSSQLToBigQuery incorrect type conversion bit-->integer

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11874:
URL: https://github.com/apache/airflow/issues/11874#issuecomment-717101888


   It looks like real problem. Can you create a PR? I'm happy to help with the review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #11874: MSSQLToBigQuery incorrect type conversion bit-->integer

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #11874:
URL: https://github.com/apache/airflow/issues/11874#issuecomment-740451441


   @danielenricocahall assigned to you


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #11874: MSSQLToBigQuery incorrect type conversion bit-->integer

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #11874:
URL: https://github.com/apache/airflow/issues/11874#issuecomment-922464690


   @danielenricocahall are you still working on this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] danielenricocahall commented on issue #11874: MSSQLToBigQuery incorrect type conversion bit-->integer

Posted by GitBox <gi...@apache.org>.
danielenricocahall commented on issue #11874:
URL: https://github.com/apache/airflow/issues/11874#issuecomment-737243187


   Not sure if anyone started working on this - can I take this issue?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org