You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/23 17:09:53 UTC
[GitHub] [airflow] e-compagno opened a new issue, #27863: SqlSensor for BigQuery fails
e-compagno opened a new issue, #27863:
URL: https://github.com/apache/airflow/issues/27863
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
apache-airflow-providers-google==8.4.0
### Apache Airflow version
2.4.2
### Operating System
Debian GNU/Linux 11 (bullseye)
### Deployment
Docker-Compose
### Deployment details
Client: Docker Engine - Community
Version: 20.10.21
Server: Docker Engine - Community
Engine:
Version: 20.10.21
API version: 1.41 (minimum version 1.12)
Docker Compose version v2.12.2
### What happened
I am trying to have a sensor task that can make queries in bigquery with the following code:
```
# BOTH IMPORT GENERATES THE SAME ERROR
#from airflow.sensors.sql import SqlSensor
from airflow.providers.common.sql.sensors.sql import SqlSensor
QUERY_CHECK = """
SELECT
CASE
WHEN COUNT(*) > 0 THEN 1
WHEN COUNT(*) = 0 THEN 0
END
FROM
`myproject.mytable`
"""
bq_sensor = SqlSensor(
task_id="bq_sensor",
conn_id="google_cloud",
#hook_kwargs={"use_legacy_sql": False},
sql=QUERY_CHECK,
mode='reschedule',
poke_interval=10,
timeout=30,
exponential_backoff=False,
soft_fail=True,
dag=dag
)
```
The error stack trace is
```
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 103, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 554, in task_test
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1752, in run
self._run_raw_task(
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 72, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1457, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1603, in _execute_task_with_callbacks
result = self._execute_task(context, task_orig)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1664, in _execute_task
result = execute_callable(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/sensors/base.py", line 189, in execute
poke_return = self.poke(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/common/sql/sensors/sql.py", line 97, in poke
hook = self._get_hook()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/common/sql/sensors/sql.py", line 90, in _get_hook
raise AirflowException(
airflow.exceptions.AirflowException: The connection type is not supported by SqlSensor. The associated hook should be a subclass of `DbApiHook`. Got GoogleBaseHook
```
### What you think should happen instead
if I define a custom operator, following [here](https://stackoverflow.com/questions/71037231/attributeerror-googlebasehook-object-has-no-attribute-get-records-error-w) as
```
from airflow.providers.common.sql.sensors.sql import SqlSensor
from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
class BigQuerySqlSensor(SqlSensor):
def _get_hook(self):
return BigQueryHook(gcp_conn_id=self.conn_id, use_legacy_sql=False, location="eu")
```
it works perfectly
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27863: SqlSensor for BigQuery fails
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27863:
URL: https://github.com/apache/airflow/issues/27863#issuecomment-1325403584
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27863: SqlSensor for BigQuery fails
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27863: SqlSensor for BigQuery fails
URL: https://github.com/apache/airflow/issues/27863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27863: SqlSensor for BigQuery fails
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27863:
URL: https://github.com/apache/airflow/issues/27863#issuecomment-1325550549
This is expected behaviour. If you want to use BigQuery Hook, you need to define your connection as BigQuery type, not google type. Conn_type should be "gcpbigquery" not "google_cloud_platform".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org