You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Daniel (Daniel Lamblin) [BDP - Seoul]" <la...@coupang.com> on 2017/11/06 02:10:46 UTC

Re: Google Sensor, last updated

I think you’d have a little more luck asking on something like stackoverflow, as it seems most devs are preoccupied by the outstanding issues and preparing a release candidate.
I’m not sure that the default ts_function is your issue, but I think that if you’re trying to use a custom one, you’ll need to implement it as a function:
https://github.com/apache/incubator-airflow/blob/3fde1043fe56fb0772d0c0f2a1b32105b18f2583/airflow/contrib/sensors/gcs_sensor.py#L102

So your
    ts_func = '{{ execution_date}}'
could be written as
    ts_func=lambda: context['execution_date']

-Daniel
On 10/31/17, 1:01 AM, "Sam Elamin" <hu...@gmail.com> wrote:

    Hi Everyone
    
    does anyone have any problems creating a
    GoogleCloudStorageObjectUpdatedSensor
    <https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/sensors/gcs_sensor.py>
    ?	
    
    
    im on airflow 1.8 and i am getting an error of
    
    *File
    "/usr/local/lib/python2.7/dist-packages/airflow/contrib/sensors/gcs_sensor.py",
    line 73, in ts_function*
    
    *[2017-10-30 15:27:10,609] {base_task_runner.py:95} INFO - Subtask: return
    context['execution_date'] + context['dag'].schedule_interval*
    
    *[2017-10-30 15:27:10,609] {base_task_runner.py:95} INFO - Subtask:
    TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'str'*
    
    *[2017-10-30 15:27:15,906] {jobs.py:2097} INFO - Task exited with return
    code 1*
    
    
    i am not overwriting the ts_function so it seems the default ts_function
    isn't working
    
    When I try to overwrite the dag with
    
    ts_fun = '{{ ds }}' or ts_func = '{{ execution_date}}'
    
    I get
    
    *''str' object is not callable'*
    
    Which I assume is because its reading the date as a string instead of a
    datetime object
    
    Any help would be greatly apprecated?
    
    
    Regards
    Sam

Re: Google Sensor, last updated

Posted by Sam Elamin <hu...@gmail.com>.
Hi Daniel

Thanks for the email. It was a bug in the contrib sensor

I raised a pr

https://github.com/apache/incubator-airflow/pull/2747

Regards
Sam

On Mon, Nov 6, 2017 at 2:10 AM, Daniel (Daniel Lamblin) [BDP - Seoul] <
lamblin@coupang.com> wrote:

> I think you’d have a little more luck asking on something like
> stackoverflow, as it seems most devs are preoccupied by the outstanding
> issues and preparing a release candidate.
> I’m not sure that the default ts_function is your issue, but I think that
> if you’re trying to use a custom one, you’ll need to implement it as a
> function:
> https://github.com/apache/incubator-airflow/blob/
> 3fde1043fe56fb0772d0c0f2a1b32105b18f2583/airflow/contrib/
> sensors/gcs_sensor.py#L102
>
> So your
>     ts_func = '{{ execution_date}}'
> could be written as
>     ts_func=lambda: context['execution_date']
>
> -Daniel
> On 10/31/17, 1:01 AM, "Sam Elamin" <hu...@gmail.com> wrote:
>
>     Hi Everyone
>
>     does anyone have any problems creating a
>     GoogleCloudStorageObjectUpdatedSensor
>     <https://github.com/apache/incubator-airflow/blob/master/
> airflow/contrib/sensors/gcs_sensor.py>
>     ?
>
>
>     im on airflow 1.8 and i am getting an error of
>
>     *File
>     "/usr/local/lib/python2.7/dist-packages/airflow/contrib/
> sensors/gcs_sensor.py",
>     line 73, in ts_function*
>
>     *[2017-10-30 15:27:10,609] {base_task_runner.py:95} INFO - Subtask:
> return
>     context['execution_date'] + context['dag'].schedule_interval*
>
>     *[2017-10-30 15:27:10,609] {base_task_runner.py:95} INFO - Subtask:
>     TypeError: unsupported operand type(s) for +: 'datetime.datetime' and
> 'str'*
>
>     *[2017-10-30 15:27:15,906] {jobs.py:2097} INFO - Task exited with
> return
>     code 1*
>
>
>     i am not overwriting the ts_function so it seems the default
> ts_function
>     isn't working
>
>     When I try to overwrite the dag with
>
>     ts_fun = '{{ ds }}' or ts_func = '{{ execution_date}}'
>
>     I get
>
>     *''str' object is not callable'*
>
>     Which I assume is because its reading the date as a string instead of a
>     datetime object
>
>     Any help would be greatly apprecated?
>
>
>     Regards
>     Sam
>
>
>