You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mfjackson (via GitHub)" <gi...@apache.org> on 2023/02/02 06:30:18 UTC
[GitHub] [airflow] mfjackson opened a new issue, #29301: BigQueryCreateEmptyTableOperator `exists_ok` parameter doesn't throw appropriate error when set to "False"
mfjackson opened a new issue, #29301:
URL: https://github.com/apache/airflow/issues/29301
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
I'm using `apache-airflow-providers-google==8.2.0`, but it looks like the relevant code that's causing this to occur is still in use as of `8.8.0`.
### Apache Airflow version
2.3.2
### Operating System
Debian (from Docker image `apache/airflow:2.3.2-python3.10`)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Deployed on an EKS cluster via Helm.
### What happened
The first task in one of my DAGs is to create an empty BigQuery table using the `BigQueryCreateEmptyTableOperator` as follows:
```python
create_staging_table = BigQueryCreateEmptyTableOperator(
task_id="create_staging_table",
dataset_id="my_dataset",
table_id="tmp_table",
schema_fields=[
{"name": "field_1", "type": "TIMESTAMP", "mode": "NULLABLE"},
{"name": "field_2", "type": "INTEGER", "mode": "NULLABLE"},
{"name": "field_3", "type": "INTEGER", "mode": "NULLABLE"}
],
exists_ok=False
)
```
Note that `exists_ok=False` explicitly here, but it is also the default value.
This task exists with a `SUCCESS` status even when `my_dataset.tmp_table` already exists in a given BigQuery project. The task returns the following logs:
```
[2023-02-02, 05:52:29 UTC] {bigquery.py:875} INFO - Creating table
[2023-02-02, 05:52:29 UTC] {bigquery.py:901} INFO - Table my_dataset.tmp_table already exists.
[2023-02-02, 05:52:30 UTC] {taskinstance.py:1395} INFO - Marking task as SUCCESS. dag_id=my_fake_dag, task_id=create_staging_table, execution_date=20230202T044000, start_date=20230202T055229, end_date=20230202T055230
[2023-02-02, 05:52:30 UTC] {local_task_job.py:156} INFO - Task exited with return code 0
```
### What you think should happen instead
Setting `exists_ok=False` should raise an exception and exit the task with a `FAILED` status if the table being created already exists in BigQuery.
### How to reproduce
1. Deploy Airflow 2.3.2 running Python 3.10 in some capacity
2. Ensure `apache-airflow-providers-google==8.2.0` (or 8.8.0, as I don't believe the issue has been fixed) is installed on the deployment.
3. Set up a GCP project and create a BigQuery dataset.
4. Create an empty BigQuery table with a schema.
5. Create a DAG that uses the `BigQueryCreateEmptyTableOperator` to create a new BigQuery table.
6. Run the DAG from Step 5 on the Airflow instance deployed in Step 1.
7. Observe the task's status.
### Anything else
I believe the silent failure may be occurring [here](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/operators/bigquery.py#L1377), as the `except` statement results in a log output, but doesn't actually raise an exception or change a state that would make the task fail.
If this is in fact the case, I'd be happy to submit a PR, but appreciate any input as to any error-handling standards/consistencies that this provider package maintains.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mfjackson commented on issue #29301: BigQueryCreateEmptyTableOperator `exists_ok` parameter doesn't throw appropriate error when set to "False"
Posted by "mfjackson (via GitHub)" <gi...@apache.org>.
mfjackson commented on issue #29301:
URL: https://github.com/apache/airflow/issues/29301#issuecomment-1426837952
@hussein-awala thanks for submitting that! I'll try to find some time over this weekend to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #29301: BigQueryCreateEmptyTableOperator `exists_ok` parameter doesn't throw appropriate error when set to "False"
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #29301:
URL: https://github.com/apache/airflow/issues/29301#issuecomment-1419914453
@mfjackson I just created this draft PR #29394 (it needs some unit tests) which creates a new param `if_exists` and deprecate `exists_ok`, this new param support 4 different actions to handle the conflict error:
- ignore: ignore the error (similar to provide `exists_ok=True`)
- log: log an error msg and success the TI (similar to provide `exists_ok=False`)
- Fail: fail the TI
- skip: mark the TI as skipped
Can you please check if it can solve your problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #29301: BigQueryCreateEmptyTableOperator `exists_ok` parameter doesn't throw appropriate error when set to "False"
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #29301: BigQueryCreateEmptyTableOperator `exists_ok` parameter doesn't throw appropriate error when set to "False"
URL: https://github.com/apache/airflow/issues/29301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org