You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/17 17:01:23 UTC

[GitHub] [airflow] owlphi opened a new issue #13139: Insert all method is not working in the bigquery hook

owlphi opened a new issue #13139:
URL: https://github.com/apache/airflow/issues/13139


   # Problem 
   
   When I am trying to use the method `insert_all` from bigquery hook I got:
    
   ```
   ValueError: Could not determine schema for table 'Table(TableReference(DatasetReference(...))
   ```
   
   The reason is that we are invoking the client insert_rows with a table without schema and the client is expecting a table with schema. 
   
   Client expecting schema in the method insert_rows [here](https://github.com/googleapis/python-bigquery/blob/master/google/cloud/bigquery/client.py#L2929), otherwise it will raise exception
   
   However we are giving a table without schema [code](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/bigquery.py#L879-L883 )
   
   ```
           table = self._resolve_table_reference(
               table_resource={}, project_id=project_id, dataset_id=dataset_id, table_id=table_id
           )
   ```
   
   # Solution proposed: 
   
   1. Get the table in bigquery 
   
   ```
   table_ref = TableReference(dataset_ref=DatasetReference(project_id, dataset_id), table_id=table_id)
   table = self.get_client(project_id=project_id).get_table(table_ref)
   ```
   
   2. Pass the table in the method insert_rows (client)
   ```
   self.get_client().insert_rows(
               table=table,
               rows=rows,
               ignore_unknown_values=ignore_unknown_values,
               skip_invalid_rows=skip_invalid_rows,
           )
   ```
           


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] owlphi commented on issue #13139: Insert all method is not working in the bigquery hook

Posted by GitBox <gi...@apache.org>.
owlphi commented on issue #13139:
URL: https://github.com/apache/airflow/issues/13139#issuecomment-747569315


   I have a PR: https://github.com/apache/airflow/pull/13138


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #13139: BigQuery hook: Insert_all (not working)

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #13139:
URL: https://github.com/apache/airflow/issues/13139


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #13139: BigQuery hook: Insert_all (not working)

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #13139:
URL: https://github.com/apache/airflow/issues/13139#issuecomment-919973409


   fixed in https://github.com/apache/airflow/pull/13138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org