Registering a plugin (sensor,hook,operator) without restarting web server and scheduler

[Sunil Khaire <su...@gmail.com>]

Hi Team ,

Is there any way to register plugins (sensors,hooks,operators)  in airflow
without even restarting the web server and scheduler.

It is difficult to take scheduler and web server down in production when
workflows are actually running everyday at a tight schedule.


--Thanks,
Sunil Khaire
Data Engineer.

Re: Registering a plugin (sensor,hook,operator) without restarting web server and scheduler

[Ash Berlin-Taylor <as...@apache.org>]

See https://www.astronomer.io/guides/airflow-importing-custom-hooks-operators/ for a guide on how to do it. 

(That uses the plugins folder as it is already in the python path, but is not using the plug-in mechanism l

-ash

On 5 June 2020 08:11:03 BST, Sunil Khaire <su...@gmail.com> wrote:
> Let me try this. Thanks for your time.
>
>Regards ,
>Sunil Khaire
>
>On Fri, Jun 5, 2020 at 12:27 PM Daniel Standish <dp...@gmail.com>
>wrote:
>
>> For these kinds of objects, you don't need to use the plugin system.
>>
>> Put your hooks / operators / sensors in a folder.  Ensure the parent
>of
>> that folder is in PYTHONPATH.  Then, in your dag files, import these
>> objects directly.
>>
>> You can use git to deploy new code to your server, for example.  And
>no
>> need to restart anything.
>>
>> On Thu, Jun 4, 2020 at 11:45 PM Sunil Khaire
><su...@gmail.com>
>> wrote:
>>
>>> Hi Team ,
>>>
>>> Is there any way to register plugins (sensors,hooks,operators)  in
>>> airflow without even restarting the web server and scheduler.
>>>
>>> It is difficult to take scheduler and web server down in production
>when
>>> workflows are actually running everyday at a tight schedule.
>>>
>>>
>>> --Thanks,
>>> Sunil Khaire
>>> Data Engineer.
>>>
>>

Re: Registering a plugin (sensor,hook,operator) without restarting web server and scheduler

[Sunil Khaire <su...@gmail.com>]

 Let me try this. Thanks for your time.

Regards ,
Sunil Khaire

On Fri, Jun 5, 2020 at 12:27 PM Daniel Standish <dp...@gmail.com>
wrote:

> For these kinds of objects, you don't need to use the plugin system.
>
> Put your hooks / operators / sensors in a folder.  Ensure the parent of
> that folder is in PYTHONPATH.  Then, in your dag files, import these
> objects directly.
>
> You can use git to deploy new code to your server, for example.  And no
> need to restart anything.
>
> On Thu, Jun 4, 2020 at 11:45 PM Sunil Khaire <su...@gmail.com>
> wrote:
>
>> Hi Team ,
>>
>> Is there any way to register plugins (sensors,hooks,operators)  in
>> airflow without even restarting the web server and scheduler.
>>
>> It is difficult to take scheduler and web server down in production when
>> workflows are actually running everyday at a tight schedule.
>>
>>
>> --Thanks,
>> Sunil Khaire
>> Data Engineer.
>>
>

Re: Registering a plugin (sensor,hook,operator) without restarting web server and scheduler

[Daniel Standish <dp...@gmail.com>]

For these kinds of objects, you don't need to use the plugin system.

Put your hooks / operators / sensors in a folder.  Ensure the parent of
that folder is in PYTHONPATH.  Then, in your dag files, import these
objects directly.

You can use git to deploy new code to your server, for example.  And no
need to restart anything.

On Thu, Jun 4, 2020 at 11:45 PM Sunil Khaire <su...@gmail.com>
wrote:

> Hi Team ,
>
> Is there any way to register plugins (sensors,hooks,operators)  in airflow
> without even restarting the web server and scheduler.
>
> It is difficult to take scheduler and web server down in production when
> workflows are actually running everyday at a tight schedule.
>
>
> --Thanks,
> Sunil Khaire
> Data Engineer.
>