You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/07 00:18:40 UTC
[GitHub] [airflow] vchiapaikeo commented on pull request #28564: GCSToBigQueryOperator allows autodetect None and infers schema
vchiapaikeo commented on PR #28564:
URL: https://github.com/apache/airflow/pull/28564#issuecomment-1374297893
Hmm I cannot repro. Can you provide a working repro of the bug?
Here is my test on newline delimited json:
file.json
```
{"col1": 2, "col2": "a"}
{"col1": 3, "col2": "b"}
{"col1": 5, "col2": "c"}
{"col1": 10, "col2": "d"}
```
gsutil cp file.json gs://my-bucket/vchiapaikeo/file.json
DAG:
```py
from airflow import DAG
from airflow.providers.google.cloud.transfers.gcs_to_bigquery import GCSToBigQueryOperator
DEFAULT_TASK_ARGS = {
"owner": "gcp-data-platform",
"retries": 1,
"retry_delay": 10,
"start_date": "2022-08-01",
}
with DAG(
max_active_runs=1,
concurrency=2,
catchup=False,
schedule_interval="@daily",
dag_id="test_os_patch_gcs_to_bigquery",
default_args=DEFAULT_TASK_ARGS,
) as dag:
test_gcs_to_bigquery_allows_autodetect_none_and_infers_schema = GCSToBigQueryOperator(
task_id="test_gcs_to_bigquery_allows_autodetect_none_and_infers_schema",
create_disposition="CREATE_IF_NEEDED",
# Need to explicitly set autodetect to None
autodetect=None,
write_disposition="WRITE_TRUNCATE",
destination_project_dataset_table="my-project.vchiapaikeo.test1",
bucket="my-bucket",
source_format="NEWLINE_DELIMITED_JSON",
source_objects=["vchiapaikeo/file.json"],
)
```
Success:
<img width="1069" alt="image" src="https://user-images.githubusercontent.com/9200263/211121042-c7359c4d-9af3-4902-9cf7-e20a5f3de44a.png">
BQ Results:
<img width="466" alt="image" src="https://user-images.githubusercontent.com/9200263/211121079-ae52ae63-2877-4f09-aa56-52fc1904d7bf.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org