[GitHub] [airflow] lewis-anderson53 commented on issue #13750: Support Standard SQL in BigQuery Sensor

[GitBox <gi...@apache.org>]

lewis-anderson53 commented on issue #13750:
URL: https://github.com/apache/airflow/issues/13750#issuecomment-797330539


   I struggled with this a while ago, I ended up creating a custom sensor specifically using the BigQuery hook that allows me to specify the parameters, very similar to the code that @omarismail94 shared.
   
   The "easy" option would be to change the default value of the [BigQueryHook](https://github.com/apache/airflow/blob/d2c2a2285c176ef232452e72a28e355667b8b50b/airflow/providers/google/cloud/hooks/bigquery.py#L67) for `use_legacy_sql` to `False` but I would guess a change like that needs some consideration. 
   I think there is justification for this: legacy SQL hasn't been the default in Google BigQuery since 2016. I'd expect 5 years on, the vast majority of BigQuery projects now use standard SQL.
   
   If a change like that isn't viable, perhaps we could look into extending the behaviour of the [Connection.get_hook()](https://github.com/apache/airflow/blob/90c15b1982dc57d258cf7cb098776e0601cb9a5c/airflow/models/connection.py#L267) method, which is called in the SqlSensor to get the hook? Right now it just returns a hook with the default params, perhaps this could be more flexible and allow the instantiation of the SqlSensor to specify custom parameter which is then transferred onto the returned hook.
   
   e.g. 
   
   ```python
   my_sensor = SqlSensor(
       ...
       hook_parameters = Dict(use_legacy_sql = False),
       ...
   )
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org