[GitHub] [airflow] BhuviTheDataGuy opened a new issue #11905: BigQueryCreateEmptyTableOperator - Not accepting the table_resource

[GitBox <gi...@apache.org>]

BhuviTheDataGuy opened a new issue #11905:
URL: https://github.com/apache/airflow/issues/11905


   In the latest BigQueryCreateEmptyTableOperator(from providers) I tried to create an empty table. And I wanted to use table resources (like creating a partition), but its not considering that.
   
   **Code to reproduce:**
   
   ```py
   from airflow import DAG
   import json
   from datetime import datetime, timedelta
   from airflow.utils.dates import days_ago
   from airflow.operators.python_operator import PythonOperator
   from airflow.providers.google.cloud.operators.bigquery import BigQueryCreateEmptyTableOperator
   
   
   # Add args and Dag
   default_args = {
       'owner': 'DBteam',
       'depends_on_past': False,
       'start_date': days_ago(1),
         'retries': 0
       }
   dag = DAG(
       'bqtest',
       default_args=default_args,
       description='create the table structure',
       schedule_interval=None,
       catchup=False,
       max_active_runs=1
   )
   resource="""{
   	"TimePartitioning": {
   		"type": "DAY",
   		"field": "created_at"
   	}
   }"""
   schema="""[{"mode": "REQUIRED", "name": "code", "type": "STRING"}, 
   {"mode": "REQUIRED", "name": "created_at", "type": "DATETIME"}, 
   {"mode": "REQUIRED", "name": "service_name", "type": "STRING"}]"""
   
   def bq_create(**kwargs):
       table_schema = 'bhuvi'
       table_name = 'sampletable'
       create = BigQueryCreateEmptyTableOperator (
       	task_id='create_bq_{}'.format(table_name),
       	project_id = 'myproject',
       	dataset_id=table_schema,
       	table_id=table_name,
       	schema_fields=json.loads(schema),
       	bigquery_conn_id='bigquery_default',
       	table_resource =json.loads(resource)
       	)
       create.execute(context=kwargs)
   
   bqcreate = PythonOperator(
   	        task_id="bqcreate",
   	        python_callable=bq_create,
   	        provide_context=True,
   	        dag=dag
   	    )
   bqcreate
   
   ```
   
   The table got created, there is no partition. (I think we have a sperate parameter for time partition, but I want to use this table resource option) 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] BhuviTheDataGuy commented on issue #11905: BigQueryCreateEmptyTableOperator - Not accepting the table_resource

[GitBox <gi...@apache.org>]

BhuviTheDataGuy commented on issue #11905:
URL: https://github.com/apache/airflow/issues/11905#issuecomment-717851729


   @turbaszek helped me to understand how this works. If I use table_resource then the rest of the parameters are ignored. 
   
   ```py
   :param table_resource: Table resource as described in documentation:
           https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#Table
           If provided all other parameters are ignored.
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] BhuviTheDataGuy closed issue #11905: BigQueryCreateEmptyTableOperator - Not accepting the table_resource

[GitBox <gi...@apache.org>]

BhuviTheDataGuy closed issue #11905:
URL: https://github.com/apache/airflow/issues/11905


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org