BigQuery Sensor Operator

[Andrew Maguire <an...@gmail.com>]

Hi,

I'm just wondering if anyone knows if there is a such thing as a sensor
operator for bigquery?

I'm trying to just trigger the dag when our daily [ga_sessions_yyyymmdd]
table we get from google analytics grows.

Not sure what the best way to do this would be.

Other options i think might be to just use the bq command line tool and a
bash based approach. Or i could have a little python script that does the
sensing or checking if table size has changed.

Any suggestions as to what a good approach might be would be greatly
appreciated.

At the moment i just run the job every few hours regardless which obviously
is not very efficient.

Cheers,
Andy

Re: BigQuery Sensor Operator

[Gerard Toonstra <gt...@gmail.com>]

There is an api where you can get table details in python. There are multiple apis all using the underlying rest one. The one i talk about is where you can call exists and get rownum and create and modified details. Saves some money and time perhaps.

Sent from my iPhone

> On 5 Jul 2017, at 12:06, Andrew Maguire <an...@gmail.com> wrote:
> 
> Hi,
> 
> I'm just wondering if anyone knows if there is a such thing as a sensor
> operator for bigquery?
> 
> I'm trying to just trigger the dag when our daily [ga_sessions_yyyymmdd]
> table we get from google analytics grows.
> 
> Not sure what the best way to do this would be.
> 
> Other options i think might be to just use the bq command line tool and a
> bash based approach. Or i could have a little python script that does the
> sensing or checking if table size has changed.
> 
> Any suggestions as to what a good approach might be would be greatly
> appreciated.
> 
> At the moment i just run the job every few hours regardless which obviously
> is not very efficient.
> 
> Cheers,
> Andy