You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ashokballolli (via GitHub)" <gi...@apache.org> on 2023/09/05 19:34:38 UTC
[GitHub] [airflow] ashokballolli opened a new issue, #34117: BigQueryInsertJobOperator doesn't log the query its executed
ashokballolli opened a new issue, #34117:
URL: https://github.com/apache/airflow/issues/34117
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Version: 2.2.3+composer
The BigQueryInsertJobOperator from airflow.providers.google.cloud.operators.bigquery takes the query to execute as part of the parameter configuration.
When it completed the execution of the task, I am not able to see the query its executed in the log like any other operator.
```
insert_query_job = BigQueryInsertJobOperator(
task_id="insert_query_job",
configuration={
"query": {
"query": INSERT_ROWS_QUERY,
"useLegacySql": False,
"priority": "BATCH",
}
},
location=location,
)
```
This task doesn't print the INSERT_ROWS_QUERY in the logs
### What you think should happen instead
It would be great for debugging if this operator prints the query its executed replacing the JINJA variables.
### How to reproduce
Run the task with query to insert the data into BigQuery table and check the logs from the logs tab on Airflow UI
### Operating System
Google Cloud Composer
### Versions of Apache Airflow Providers
2.2.3+composer
But I guess this is valid for all the versions of Airflow
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashokballolli closed issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "ashokballolli (via GitHub)" <gi...@apache.org>.
ashokballolli closed issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
URL: https://github.com/apache/airflow/issues/34117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34117:
URL: https://github.com/apache/airflow/issues/34117#issuecomment-1707203429
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashokballolli commented on issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "ashokballolli (via GitHub)" <gi...@apache.org>.
ashokballolli commented on issue #34117:
URL: https://github.com/apache/airflow/issues/34117#issuecomment-1707214049
Its not a blocker as I am able to see the query executed as part of the `Rendered Template` tab in the **Airflow UI**. But it would be great to log it so that if someone is only checking the GCP's Cloud Logging/stackdriver(?) then it really helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #34117:
URL: https://github.com/apache/airflow/issues/34117#issuecomment-1707831773
It seems there's already a log long ago. https://github.com/apache/airflow/blob/7731255975b9ac1c8cf7b2e363aaa8b9cbba80b8/airflow/providers/google/cloud/operators/bigquery.py#L2799. Please let us know if I misunderstand anything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashokballolli commented on issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "ashokballolli (via GitHub)" <gi...@apache.org>.
ashokballolli commented on issue #34117:
URL: https://github.com/apache/airflow/issues/34117#issuecomment-1709226919
Hi @Lee-W @nathadfield thank you very much for checking this. Yes, our GCP Cloud Composer is running on a old version, and the latest available version to deploy through Composer is 2.2.3 and this change I guess is still not available in this version.
We will wait to upgrade to the latest version as soon as its available.
Thank you very much for your time.
This ticket is good to close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #34117:
URL: https://github.com/apache/airflow/issues/34117#issuecomment-1708630879
@Lee-W @ashokballolli In at least Google provider v10.6.0, what you see in the log is the output of the whole configuration object. e.g.
```
test = BigQueryInsertJobOperator(
task_id='test',
configuration={
'query': {
'query': 'select true',
'useLegacySql': False,
}
}
)
```
```
[2023-09-06, 15:33:05 UTC] {bigquery.py:2792} INFO - Executing: {'query': {'query': 'select true', 'useLegacySql': False}}'
```
So, it does contain the SQL although it is perhaps not that easy to isolate. Not sure that warrants an additional logging step to specifically pick out `query` though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Lee-W commented on issue #34117: BigQueryInsertJobOperator doesn't log the query its executed
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #34117:
URL: https://github.com/apache/airflow/issues/34117#issuecomment-1707819818
Hi @ashokballolli , by 2.3.3, do you mean the airflow version? or [apache-airflow-providers-google](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html) version? To investigate issues on providers like this one, I think we'll need the provider version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org