You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/06 18:28:11 UTC
[GitHub] [airflow] andreypanchenko opened a new issue #20081: BigQueryExecuteQueryOperator timeout issue
andreypanchenko opened a new issue #20081:
URL: https://github.com/apache/airflow/issues/20081
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==6.1.0
### Apache Airflow version
2.2.2 (latest released)
### Operating System
NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal
### Deployment
Virtualenv installation
### Deployment details
_No response_
### What happened
Hi colleagues, faced with an issue in Airflow. In short:
I trying to move from 1.14 version to 2.2.2 ver. However when I wan't substitute BigQueryOperator to BigQueryExecuteQueryOperator
I tried to increase jobTimeoutMs how [Bigquery API](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query) docs are told but still see this issue.
```
mobile_push_stat = BigQueryExecuteQueryOperator(
task_id="mobile_push_stat",
sql="/sql/updater/mobile_push_stat.sql",
use_legacy_sql=False,
api_resource_configs={"jobTimeoutMs": "3600000"},
gcp_conn_id="bigquery_work",
)
```
### What you expected to happen
I can't prolong timeout for longterm job executionlong-term
### How to reproduce
You should use a query that can be running along for more than one minute.
### Anything else
```
*** Reading local file: /srv/airflow/logs/ExtensionPushStat/mobile_push_stat/2021-12-05T06:00:00+00:00/29.log
[2021-12-06, 17:33:11 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: ExtensionPushStat.mobile_push_stat scheduled__2021-12-05T06:00:00+00:00 [queued]>
[2021-12-06, 17:33:11 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: ExtensionPushStat.mobile_push_stat scheduled__2021-12-05T06:00:00+00:00 [queued]>
[2021-12-06, 17:33:11 UTC] {taskinstance.py:1241} INFO -
--------------------------------------------------------------------------------
[2021-12-06, 17:33:11 UTC] {taskinstance.py:1242} INFO - Starting attempt 29 of 29
[2021-12-06, 17:33:11 UTC] {taskinstance.py:1243} INFO -
--------------------------------------------------------------------------------
[2021-12-06, 17:33:11 UTC] {taskinstance.py:1262} INFO - Executing <Task(BigQueryExecuteQueryOperator): mobile_push_stat> on 2021-12-05 06:00:00+00:00
[2021-12-06, 17:33:11 UTC] {base_task_runner.py:141} INFO - Running on host: airflow-vm-v2
[2021-12-06, 17:33:11 UTC] {base_task_runner.py:142} INFO - Running: ['airflow', 'tasks', 'run', 'ExtensionPushStat', 'mobile_push_stat', 'scheduled__2021-12-05T06:00:00+00:00', '--job-id', '3235', '--raw', '--subdir', 'DAGS_FOLDER/ExtensionPushStat.py', '--cfg-path', '/tmp/tmpk6p4qql0', '--error-file', '/tmp/tmpwz3z5o0y']
[2021-12-06, 17:33:12 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat [[34m2021-12-06, 17:33:12 UTC[0m] {[34mdagbag.py:[0m500} INFO[0m - Filling up the DagBag from /srv/airflow/dags/ExtensionPushStat.py[0m
[2021-12-06, 17:34:11 UTC] {local_task_job.py:206} WARNING - Recorded pid 137796 does not match the current pid 137817
[2021-12-06, 17:34:11 UTC] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 137817
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat Running <TaskInstance: ExtensionPushStat.mobile_push_stat scheduled__2021-12-05T06:00:00+00:00 [running]> on host airflow-vm-v2
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat Traceback (most recent call last):
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/bin/airflow", line 8, in <module>
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat sys.exit(main())
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat args.func(args)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat return func(*args, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat return f(*args, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat _run_task_by_selected_method(args, dag, ti)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat _run_raw_task(args, ti)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 180, in _run_raw_task
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat ti._run_raw_task(
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat return func(*args, session=session, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat self._execute_task_with_callbacks(context)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat result = self._execute_task(context, self.task)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat result = execute_callable(context=context)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/providers/google/cloud/operators/bigquery.py", line 693, in execute
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat job_id = self.hook.run_query(
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2325, in run_query
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat job = self.insert_job(configuration=configuration, project_id=self.project_id)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/providers/google/common/hooks/base_google.py", line 425, in inner_wrapper
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat return func(self, *args, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 1639, in insert_job
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat job.result()
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1450, in result
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat do_get_result()
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1440, in do_get_result
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat super(QueryJob, self).result(retry=retry, timeout=timeout)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/cloud/bigquery/job/base.py", line 727, in result
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/api_core/future/polling.py", line 130, in result
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat self._blocking_poll(timeout=timeout, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/cloud/bigquery/job/query.py", line 1199, in _blocking_poll
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat super(QueryJob, self)._blocking_poll(timeout=timeout, **kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/api_core/future/polling.py", line 108, in _blocking_poll
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat retry_(self._done_or_raise)(**kwargs)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat return retry_target(
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/google/api_core/retry.py", line 220, in retry_target
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat time.sleep(sleep)
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat File "/srv/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1413, in signal_handler
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat raise AirflowException("Task received SIGTERM signal")
[2021-12-06, 17:34:11 UTC] {base_task_runner.py:122} INFO - Job 3235: Subtask mobile_push_stat airflow.exceptions.AirflowException: Task received SIGTERM signal
[2021-12-06, 17:34:11 UTC] {process_utils.py:66} INFO - Process psutil.Process(pid=137817, status='terminated', exitcode=1, started='17:33:11') (137817) terminated with exit code 1
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987261565
Regarding to the documentation the parameter `api_resource_configs` seems should be like `{'query': {'useQueryCache': False, 'timeoutMs': 100000}}`
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryExecuteQueryOperator
Anyway, this operator will be deprecated, and better try to use `BigQueryInsertJobOperator` like:
```
insert_query_job = BigQueryInsertJobOperator(
task_id="mobile_push_stat",
configuration={
"query": {
"query": "{% include '/sql/updater/mobile_push_stat.sql' %}",
"useLegacySql": False,
"jobTimeoutMs": 3600000
}
},
location=location,
gcp_conn_id=gcp_conn
)
```
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryInsertJobOperator
Maybe you will find helpful these examples:
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987343959
I had a similar issue when I've installed Airflow using `pip` on VM.
This VM had few hostnames that's why tasks were killed.
I fixed it by changing `hostname_callable` in `airflow.cfg` from `socket.getfqdn` to `airflow.utils.net.get_host_ip_address`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andreypanchenko closed issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
andreypanchenko closed issue #20081:
URL: https://github.com/apache/airflow/issues/20081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987261565
Hi Andrey. Nice to see you here
Regarding to the documentation the parameter `api_resource_configs` seems should be like `{'query': {'useQueryCache': False, 'timeoutMs': 100000}}`
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#queryrequest
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryExecuteQueryOperator
Anyway, this operator will be deprecated, and better try to use `BigQueryInsertJobOperator` like:
```
insert_query_job = BigQueryInsertJobOperator(
task_id="mobile_push_stat",
configuration={
"query": {
"query": "{% include '/sql/updater/mobile_push_stat.sql' %}",
"useLegacySql": False,
},
"jobTimeoutMs": 3600000
},
location=location,
gcp_conn_id=gcp_conn
)
```
https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobconfiguration
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryInsertJobOperator
Maybe you will find helpful these examples:
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
Note that in first case `timeoutMs` and in second `jobTimeoutMs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987275825
Also in this case there is an unusual error.
There is `AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")` expected if API returns some error.
In your case, I guess some another error. Check parameter `operation_timeout` if you are using CeleryExecutor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987261565
Regarding to the documentation the parameter `api_resource_configs` seems should be like `{'query': {'useQueryCache': False, 'timeoutMs': 100000}}`
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#queryrequest
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryExecuteQueryOperator
Anyway, this operator will be deprecated, and better try to use `BigQueryInsertJobOperator` like:
```
insert_query_job = BigQueryInsertJobOperator(
task_id="mobile_push_stat",
configuration={
"query": {
"query": "{% include '/sql/updater/mobile_push_stat.sql' %}",
"useLegacySql": False,
},
"jobTimeoutMs": 3600000
},
location=location,
gcp_conn_id=gcp_conn
)
```
https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobconfiguration
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryInsertJobOperator
Maybe you will find helpful these examples:
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
Note that in first case `timeoutMs` and in second `jobTimeoutMs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987275825
Also in this case there is an unusual error.
There is `AirflowException(f"BigQuery job {job.job_id} failed: {job.error_result}")` expected if API returns some error.
In your case, I guess some another error. Check parameter `operation_timeout` if you are use CeleryExecutor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andreypanchenko commented on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
andreypanchenko commented on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987719707
Hi Dmytro Kazanzhy,
Thank you for your help yes, this is an issue that does not relate to Bigquery or Google.
This issue relate to two config parameters
killed_task_cleanup_time - in this case for long-running jobs needs to be increased
job_heartbeat_sec - in this case when you want to clear status for the failed task and re-run from UI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20081: BigQueryExecuteQueryOperator timeout issue
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987041201
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987261565
Hi Andrey. Nice to see you here
Regarding to the documentation the parameter `api_resource_configs` seems should be like `{'query': {'useQueryCache': False, 'timeoutMs': 100000}}`
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query#queryrequest
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryExecuteQueryOperator
Anyway, this operator will be deprecated, and better try to use `BigQueryInsertJobOperator` like:
```
insert_query_job = BigQueryInsertJobOperator(
task_id="mobile_push_stat",
configuration={
"query": {
"query": "{% include '/sql/updater/mobile_push_stat.sql' %}",
"useLegacySql": False,
},
"jobTimeoutMs": "3600000",
},
location=location,
gcp_conn_id=gcp_conn
)
```
https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#jobconfiguration
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryInsertJobOperator
Maybe you will find helpful these examples:
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
Note that in first case `timeoutMs` and in second `jobTimeoutMs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on issue #20081: BigQueryExecuteQueryOperator timeout issue (how to increase timeout)
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #20081:
URL: https://github.com/apache/airflow/issues/20081#issuecomment-987261565
Regarding to the documentation the parameter `api_resource_configs` seems should be like `{'query': {'useQueryCache': False, 'timeoutMs': 100000}}`
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryExecuteQueryOperator
Anyway, this operator will be deprecated, and better try to use `BigQueryInsertJobOperator` like:
```
insert_query_job = BigQueryInsertJobOperator(
task_id="mobile_push_stat",
configuration={
"query": {
"query": "{% include '/sql/updater/mobile_push_stat.sql' %}",
"useLegacySql": False,
},
"jobTimeoutMs": 3600000
},
location=location,
gcp_conn_id=gcp_conn
)
```
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_modules/airflow/providers/google/cloud/operators/bigquery.html#BigQueryInsertJobOperator
Maybe you will find helpful these examples:
https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org