You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Pedro Machado <pe...@205datalab.com> on 2018/05/26 00:48:16 UTC

HttpSensor raising exception with status=403

Hi,

I am using HttpSensor to look for a file. The webserver is returning 403
(instead of 404) while the file is not available. This is causing the
sensor to raise an exception.

I see that a recent commit added the ability to disable the call to
response.raise_for_status() on the http hook by passing
extra_options={'check_response': False} to the sensor.

https://github.com/apache/incubator-airflow/commit/6c19468e0b3b938249acc43e4b833a753d093efc?diff=unified

I am unable to upgrade airflow. What would be the best way to incorporate
the new code, perhaps into a custom sensor?

Thanks,

Pedro

Re: HttpSensor raising exception with status=403

Posted by Pedro Machado <pe...@205datalab.com>.

Thanks! I ended up creating a plugin and it's working OK.


On Mon, May 28, 2018 at 9:22 AM Driesprong, Fokko <fo...@driesprong.frl>
wrote:

> Hi Pedro,
>
> You could just create a CustomHttpHook and place it on your pythonpath,
> then you should also create a CustomHttpSensor. Hope this helps.
>
> Cheers, Fokko
>
> 2018-05-26 2:48 GMT+02:00 Pedro Machado <pe...@205datalab.com>:
>
> > Hi,
> >
> > I am using HttpSensor to look for a file. The webserver is returning 403
> > (instead of 404) while the file is not available. This is causing the
> > sensor to raise an exception.
> >
> > I see that a recent commit added the ability to disable the call to
> > response.raise_for_status() on the http hook by passing
> > extra_options={'check_response': False} to the sensor.
> >
> > https://github.com/apache/incubator-airflow/commit/
> > 6c19468e0b3b938249acc43e4b833a753d093efc?diff=unified
> >
> > I am unable to upgrade airflow. What would be the best way to incorporate
> > the new code, perhaps into a custom sensor?
> >
> > Thanks,
> >
> > Pedro
> >
>

Re: HttpSensor raising exception with status=403

Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.

Hi Pedro,

You could just create a CustomHttpHook and place it on your pythonpath,
then you should also create a CustomHttpSensor. Hope this helps.

Cheers, Fokko

2018-05-26 2:48 GMT+02:00 Pedro Machado <pe...@205datalab.com>:

> Hi,
>
> I am using HttpSensor to look for a file. The webserver is returning 403
> (instead of 404) while the file is not available. This is causing the
> sensor to raise an exception.
>
> I see that a recent commit added the ability to disable the call to
> response.raise_for_status() on the http hook by passing
> extra_options={'check_response': False} to the sensor.
>
> https://github.com/apache/incubator-airflow/commit/
> 6c19468e0b3b938249acc43e4b833a753d093efc?diff=unified
>
> I am unable to upgrade airflow. What would be the best way to incorporate
> the new code, perhaps into a custom sensor?
>
> Thanks,
>
> Pedro
>