You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kyle Hamlin <ha...@gmail.com> on 2018/03/29 22:56:35 UTC
What are the advantages of plugins, not sure I see any?
Hello,
I just got done writing a few plugins, and the process has left me
wondering what the real benefits are? As far as I can tell, it makes
testing more difficult since you cannot import from the created module, you
have to import directly from the plugin. Additionally, your code editor
isn't aware of these new plugin modules since they are created when you
start the app up, this makes it seem like there are errors when there
aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
inside and be done with it? Very curious what peoples thoughts are, who
knows I could be testing wrong or writing the plugins wrong. Thanks in
advance!
Re: What are the advantages of plugins, not sure I see any?
Posted by James Meickle <jm...@quantopian.com>.
I have internally been working on a stock market date/time operator plugin
that I am hoping to open source soon. Figuring out the best way to combine
Airflow plugin packaging and Python packaging has been rather unpleasant,
to be honest. I'd love to see this revisited in a future version of Airflow
so that plugins are just a layer on top of standard Python packages.
On Fri, Mar 30, 2018 at 4:53 PM, Taylor Edmiston <te...@gmail.com>
wrote:
> We might be an edge case running Airflow as a platform at Astronomer, but
> we make hooks and operators that are reused across many Airflow instances
> by customers. (Starting to open source at
> https://github.com/airflow-plugins.) We've also run a Mesos executor as a
> plugin for similar reasons so that as we fix bugs or add features we can
> reuse it across Airflow installs.
>
> To add one more point — we've tossed around the idea of building more
> tooling around plugins, so you could do something like: $ airflow plugin
> install -U github-plugin then import GitHubHook and...
>
> A package install via PyPI could work as well but we haven't seen anyone
> else distributing Airflow plugins as packages yet.
>
> T
>
> *Taylor Edmiston*
> TEdmiston.com <https://www.tedmiston.com/> | Blog
> <http://blog.tedmiston.com>
> Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor>
>
>
> On Fri, Mar 30, 2018 at 4:37 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Yes, it makes most sense to just have a `common/hooks` and
> > `common/operators/` in the repo where your DAGs live and import them at
> > will.
> >
> > Max
> >
> > On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <ha...@gmail.com>
> wrote:
> >
> > > Thanks for the responses! I think my conclusion was similar, they seem
> > good
> > > for redistribution, but if you're only working with operators and hooks
> > and
> > > aren't sharing that code then it might not make too much sense to use
> > them.
> > >
> > > On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > > > The original intent was to use plugins as a way to share sets of
> > objects
> > > > and applications build on top of Airflow.
> > > >
> > > > For instance it'd be possible to ship the things listed bellow as
> > Airflow
> > > > plugin:
> > > >
> > > > * "validate-and-schedule my query" UI
> > > > * a set of ML-related hooks and operators that match a specific
> > workflow
> > > > * a Hive metastore-browser UI
> > > > * drag and drop UI
> > > > * ...
> > > >
> > > > As far as I know it's not a super popular feature. Maybe the scope of
> > > > Airflow is already large enough without having all that stuff sitting
> > on
> > > > top of it. :)
> > > >
> > > > As George pointed out it could also allow to accelerate the release
> > > cadence
> > > > of sets of things that are currently inside Airflow. Things like
> Google
> > > > Cloud-related operators and hooks could ship as a plugin on their own
> > > > release schedule.
> > > >
> > > > Max
> > > >
> > > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > > > <(949)%20412-7220> <
> > > > alex.n.james@gmail.com> wrote:
> > > >
> > > > > At Netflix we've put our plugin inside the DAGs folder and pointed
> > the
> > > > > config to it there so we can both import directly in DAGs AND
> update
> > > the
> > > > > plugin as we go. This makes it easy to test changes to operators
> > needed
> > > > for
> > > > > ongoing DAG development in the same PR.
> > > > >
> > > > > The two plugin features I've used which don't translate to the
> direct
> > > > > import approach are custom macros (we provide some internal
> > libraries)
> > > > and
> > > > > UI menu links, which we use for linking local docs describing our
> > > > > deployment and custom operators, server/worker monitoring with
> atlas,
> > > and
> > > > > genie job monitoring.
> > > > >
> > > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > > > > <ge...@cloverhealth.com.invalid> wrote:
> > > > >
> > > > > > It's presumably useful if you want to package your plugins for
> > other
> > > > > people
> > > > > > to use but it seems like everyone just adds those directly to the
> > > > Airflow
> > > > > > codebase these days.
> > > > > >
> > > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <hamlin.kn@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Yeah so far I have only written hooks and operators so maybe
> the
> > > > > benefit
> > > > > > > only kicks in for other airflow abstractions.
> > > > > > >
> > > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > > > george@cloverhealth.com.INVALID> wrote:
> > > > > > > >
> > > > > > > > We also import our operators and sensors directly.
> > > > > > > >
> > > > > > > > However, executors and some other pieces are a little bit
> > harder
> > > to
> > > > > > deal
> > > > > > > > with as non-plugins
> > > > > > > >
> > > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> > > hamlin.kn@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hello,
> > > > > > > >>
> > > > > > > >> I just got done writing a few plugins, and the process has
> > left
> > > me
> > > > > > > >> wondering what the real benefits are? As far as I can tell,
> it
> > > > makes
> > > > > > > >> testing more difficult since you cannot import from the
> > created
> > > > > > module,
> > > > > > > you
> > > > > > > >> have to import directly from the plugin. Additionally, your
> > code
> > > > > > editor
> > > > > > > >> isn't aware of these new plugin modules since they are
> created
> > > > when
> > > > > > you
> > > > > > > >> start the app up, this makes it seem like there are errors
> > when
> > > > > there
> > > > > > > >> aren't. Why not just create a lib/ dir with hooks, operators
> > > etc..
> > > > > > dirs
> > > > > > > >> inside and be done with it? Very curious what peoples
> thoughts
> > > > are,
> > > > > > who
> > > > > > > >> knows I could be testing wrong or writing the plugins wrong.
> > > > Thanks
> > > > > in
> > > > > > > >> advance!
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: What are the advantages of plugins, not sure I see any?
Posted by Taylor Edmiston <te...@gmail.com>.
We might be an edge case running Airflow as a platform at Astronomer, but
we make hooks and operators that are reused across many Airflow instances
by customers. (Starting to open source at
https://github.com/airflow-plugins.) We've also run a Mesos executor as a
plugin for similar reasons so that as we fix bugs or add features we can
reuse it across Airflow installs.
To add one more point — we've tossed around the idea of building more
tooling around plugins, so you could do something like: $ airflow plugin
install -U github-plugin then import GitHubHook and...
A package install via PyPI could work as well but we haven't seen anyone
else distributing Airflow plugins as packages yet.
T
*Taylor Edmiston*
TEdmiston.com <https://www.tedmiston.com/> | Blog
<http://blog.tedmiston.com>
Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor>
On Fri, Mar 30, 2018 at 4:37 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> Yes, it makes most sense to just have a `common/hooks` and
> `common/operators/` in the repo where your DAGs live and import them at
> will.
>
> Max
>
> On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <ha...@gmail.com> wrote:
>
> > Thanks for the responses! I think my conclusion was similar, they seem
> good
> > for redistribution, but if you're only working with operators and hooks
> and
> > aren't sharing that code then it might not make too much sense to use
> them.
> >
> > On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > The original intent was to use plugins as a way to share sets of
> objects
> > > and applications build on top of Airflow.
> > >
> > > For instance it'd be possible to ship the things listed bellow as
> Airflow
> > > plugin:
> > >
> > > * "validate-and-schedule my query" UI
> > > * a set of ML-related hooks and operators that match a specific
> workflow
> > > * a Hive metastore-browser UI
> > > * drag and drop UI
> > > * ...
> > >
> > > As far as I know it's not a super popular feature. Maybe the scope of
> > > Airflow is already large enough without having all that stuff sitting
> on
> > > top of it. :)
> > >
> > > As George pointed out it could also allow to accelerate the release
> > cadence
> > > of sets of things that are currently inside Airflow. Things like Google
> > > Cloud-related operators and hooks could ship as a plugin on their own
> > > release schedule.
> > >
> > > Max
> > >
> > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > > <(949)%20412-7220> <
> > > alex.n.james@gmail.com> wrote:
> > >
> > > > At Netflix we've put our plugin inside the DAGs folder and pointed
> the
> > > > config to it there so we can both import directly in DAGs AND update
> > the
> > > > plugin as we go. This makes it easy to test changes to operators
> needed
> > > for
> > > > ongoing DAG development in the same PR.
> > > >
> > > > The two plugin features I've used which don't translate to the direct
> > > > import approach are custom macros (we provide some internal
> libraries)
> > > and
> > > > UI menu links, which we use for linking local docs describing our
> > > > deployment and custom operators, server/worker monitoring with atlas,
> > and
> > > > genie job monitoring.
> > > >
> > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > > > <ge...@cloverhealth.com.invalid> wrote:
> > > >
> > > > > It's presumably useful if you want to package your plugins for
> other
> > > > people
> > > > > to use but it seems like everyone just adds those directly to the
> > > Airflow
> > > > > codebase these days.
> > > > >
> > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <ha...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Yeah so far I have only written hooks and operators so maybe the
> > > > benefit
> > > > > > only kicks in for other airflow abstractions.
> > > > > >
> > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > > george@cloverhealth.com.INVALID> wrote:
> > > > > > >
> > > > > > > We also import our operators and sensors directly.
> > > > > > >
> > > > > > > However, executors and some other pieces are a little bit
> harder
> > to
> > > > > deal
> > > > > > > with as non-plugins
> > > > > > >
> > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> > hamlin.kn@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hello,
> > > > > > >>
> > > > > > >> I just got done writing a few plugins, and the process has
> left
> > me
> > > > > > >> wondering what the real benefits are? As far as I can tell, it
> > > makes
> > > > > > >> testing more difficult since you cannot import from the
> created
> > > > > module,
> > > > > > you
> > > > > > >> have to import directly from the plugin. Additionally, your
> code
> > > > > editor
> > > > > > >> isn't aware of these new plugin modules since they are created
> > > when
> > > > > you
> > > > > > >> start the app up, this makes it seem like there are errors
> when
> > > > there
> > > > > > >> aren't. Why not just create a lib/ dir with hooks, operators
> > etc..
> > > > > dirs
> > > > > > >> inside and be done with it? Very curious what peoples thoughts
> > > are,
> > > > > who
> > > > > > >> knows I could be testing wrong or writing the plugins wrong.
> > > Thanks
> > > > in
> > > > > > >> advance!
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: What are the advantages of plugins, not sure I see any?
Posted by Maxime Beauchemin <ma...@gmail.com>.
Yes, it makes most sense to just have a `common/hooks` and
`common/operators/` in the repo where your DAGs live and import them at
will.
Max
On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <ha...@gmail.com> wrote:
> Thanks for the responses! I think my conclusion was similar, they seem good
> for redistribution, but if you're only working with operators and hooks and
> aren't sharing that code then it might not make too much sense to use them.
>
> On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > The original intent was to use plugins as a way to share sets of objects
> > and applications build on top of Airflow.
> >
> > For instance it'd be possible to ship the things listed bellow as Airflow
> > plugin:
> >
> > * "validate-and-schedule my query" UI
> > * a set of ML-related hooks and operators that match a specific workflow
> > * a Hive metastore-browser UI
> > * drag and drop UI
> > * ...
> >
> > As far as I know it's not a super popular feature. Maybe the scope of
> > Airflow is already large enough without having all that stuff sitting on
> > top of it. :)
> >
> > As George pointed out it could also allow to accelerate the release
> cadence
> > of sets of things that are currently inside Airflow. Things like Google
> > Cloud-related operators and hooks could ship as a plugin on their own
> > release schedule.
> >
> > Max
> >
> > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > <(949)%20412-7220> <
> > alex.n.james@gmail.com> wrote:
> >
> > > At Netflix we've put our plugin inside the DAGs folder and pointed the
> > > config to it there so we can both import directly in DAGs AND update
> the
> > > plugin as we go. This makes it easy to test changes to operators needed
> > for
> > > ongoing DAG development in the same PR.
> > >
> > > The two plugin features I've used which don't translate to the direct
> > > import approach are custom macros (we provide some internal libraries)
> > and
> > > UI menu links, which we use for linking local docs describing our
> > > deployment and custom operators, server/worker monitoring with atlas,
> and
> > > genie job monitoring.
> > >
> > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > > <ge...@cloverhealth.com.invalid> wrote:
> > >
> > > > It's presumably useful if you want to package your plugins for other
> > > people
> > > > to use but it seems like everyone just adds those directly to the
> > Airflow
> > > > codebase these days.
> > > >
> > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <ha...@gmail.com>
> > wrote:
> > > >
> > > > > Yeah so far I have only written hooks and operators so maybe the
> > > benefit
> > > > > only kicks in for other airflow abstractions.
> > > > >
> > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > george@cloverhealth.com.INVALID> wrote:
> > > > > >
> > > > > > We also import our operators and sensors directly.
> > > > > >
> > > > > > However, executors and some other pieces are a little bit harder
> to
> > > > deal
> > > > > > with as non-plugins
> > > > > >
> > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> hamlin.kn@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Hello,
> > > > > >>
> > > > > >> I just got done writing a few plugins, and the process has left
> me
> > > > > >> wondering what the real benefits are? As far as I can tell, it
> > makes
> > > > > >> testing more difficult since you cannot import from the created
> > > > module,
> > > > > you
> > > > > >> have to import directly from the plugin. Additionally, your code
> > > > editor
> > > > > >> isn't aware of these new plugin modules since they are created
> > when
> > > > you
> > > > > >> start the app up, this makes it seem like there are errors when
> > > there
> > > > > >> aren't. Why not just create a lib/ dir with hooks, operators
> etc..
> > > > dirs
> > > > > >> inside and be done with it? Very curious what peoples thoughts
> > are,
> > > > who
> > > > > >> knows I could be testing wrong or writing the plugins wrong.
> > Thanks
> > > in
> > > > > >> advance!
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: What are the advantages of plugins, not sure I see any?
Posted by Kyle Hamlin <ha...@gmail.com>.
Thanks for the responses! I think my conclusion was similar, they seem good
for redistribution, but if you're only working with operators and hooks and
aren't sharing that code then it might not make too much sense to use them.
On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> The original intent was to use plugins as a way to share sets of objects
> and applications build on top of Airflow.
>
> For instance it'd be possible to ship the things listed bellow as Airflow
> plugin:
>
> * "validate-and-schedule my query" UI
> * a set of ML-related hooks and operators that match a specific workflow
> * a Hive metastore-browser UI
> * drag and drop UI
> * ...
>
> As far as I know it's not a super popular feature. Maybe the scope of
> Airflow is already large enough without having all that stuff sitting on
> top of it. :)
>
> As George pointed out it could also allow to accelerate the release cadence
> of sets of things that are currently inside Airflow. Things like Google
> Cloud-related operators and hooks could ship as a plugin on their own
> release schedule.
>
> Max
>
> On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> <(949)%20412-7220> <
> alex.n.james@gmail.com> wrote:
>
> > At Netflix we've put our plugin inside the DAGs folder and pointed the
> > config to it there so we can both import directly in DAGs AND update the
> > plugin as we go. This makes it easy to test changes to operators needed
> for
> > ongoing DAG development in the same PR.
> >
> > The two plugin features I've used which don't translate to the direct
> > import approach are custom macros (we provide some internal libraries)
> and
> > UI menu links, which we use for linking local docs describing our
> > deployment and custom operators, server/worker monitoring with atlas, and
> > genie job monitoring.
> >
> > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > <ge...@cloverhealth.com.invalid> wrote:
> >
> > > It's presumably useful if you want to package your plugins for other
> > people
> > > to use but it seems like everyone just adds those directly to the
> Airflow
> > > codebase these days.
> > >
> > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <ha...@gmail.com>
> wrote:
> > >
> > > > Yeah so far I have only written hooks and operators so maybe the
> > benefit
> > > > only kicks in for other airflow abstractions.
> > > >
> > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > george@cloverhealth.com.INVALID> wrote:
> > > > >
> > > > > We also import our operators and sensors directly.
> > > > >
> > > > > However, executors and some other pieces are a little bit harder to
> > > deal
> > > > > with as non-plugins
> > > > >
> > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <ha...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Hello,
> > > > >>
> > > > >> I just got done writing a few plugins, and the process has left me
> > > > >> wondering what the real benefits are? As far as I can tell, it
> makes
> > > > >> testing more difficult since you cannot import from the created
> > > module,
> > > > you
> > > > >> have to import directly from the plugin. Additionally, your code
> > > editor
> > > > >> isn't aware of these new plugin modules since they are created
> when
> > > you
> > > > >> start the app up, this makes it seem like there are errors when
> > there
> > > > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> > > dirs
> > > > >> inside and be done with it? Very curious what peoples thoughts
> are,
> > > who
> > > > >> knows I could be testing wrong or writing the plugins wrong.
> Thanks
> > in
> > > > >> advance!
> > > > >>
> > > >
> > >
> >
>
Re: What are the advantages of plugins, not sure I see any?
Posted by Maxime Beauchemin <ma...@gmail.com>.
The original intent was to use plugins as a way to share sets of objects
and applications build on top of Airflow.
For instance it'd be possible to ship the things listed bellow as Airflow
plugin:
* "validate-and-schedule my query" UI
* a set of ML-related hooks and operators that match a specific workflow
* a Hive metastore-browser UI
* drag and drop UI
* ...
As far as I know it's not a super popular feature. Maybe the scope of
Airflow is already large enough without having all that stuff sitting on
top of it. :)
As George pointed out it could also allow to accelerate the release cadence
of sets of things that are currently inside Airflow. Things like Google
Cloud-related operators and hooks could ship as a plugin on their own
release schedule.
Max
On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220 <
alex.n.james@gmail.com> wrote:
> At Netflix we've put our plugin inside the DAGs folder and pointed the
> config to it there so we can both import directly in DAGs AND update the
> plugin as we go. This makes it easy to test changes to operators needed for
> ongoing DAG development in the same PR.
>
> The two plugin features I've used which don't translate to the direct
> import approach are custom macros (we provide some internal libraries) and
> UI menu links, which we use for linking local docs describing our
> deployment and custom operators, server/worker monitoring with atlas, and
> genie job monitoring.
>
> On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> <ge...@cloverhealth.com.invalid> wrote:
>
> > It's presumably useful if you want to package your plugins for other
> people
> > to use but it seems like everyone just adds those directly to the Airflow
> > codebase these days.
> >
> > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <ha...@gmail.com> wrote:
> >
> > > Yeah so far I have only written hooks and operators so maybe the
> benefit
> > > only kicks in for other airflow abstractions.
> > >
> > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > george@cloverhealth.com.INVALID> wrote:
> > > >
> > > > We also import our operators and sensors directly.
> > > >
> > > > However, executors and some other pieces are a little bit harder to
> > deal
> > > > with as non-plugins
> > > >
> > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <ha...@gmail.com>
> > > wrote:
> > > >>
> > > >> Hello,
> > > >>
> > > >> I just got done writing a few plugins, and the process has left me
> > > >> wondering what the real benefits are? As far as I can tell, it makes
> > > >> testing more difficult since you cannot import from the created
> > module,
> > > you
> > > >> have to import directly from the plugin. Additionally, your code
> > editor
> > > >> isn't aware of these new plugin modules since they are created when
> > you
> > > >> start the app up, this makes it seem like there are errors when
> there
> > > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> > dirs
> > > >> inside and be done with it? Very curious what peoples thoughts are,
> > who
> > > >> knows I could be testing wrong or writing the plugins wrong. Thanks
> in
> > > >> advance!
> > > >>
> > >
> >
>
Re: What are the advantages of plugins, not sure I see any?
Posted by Alex Tronchin-James 949-412-7220 <al...@gmail.com>.
At Netflix we've put our plugin inside the DAGs folder and pointed the
config to it there so we can both import directly in DAGs AND update the
plugin as we go. This makes it easy to test changes to operators needed for
ongoing DAG development in the same PR.
The two plugin features I've used which don't translate to the direct
import approach are custom macros (we provide some internal libraries) and
UI menu links, which we use for linking local docs describing our
deployment and custom operators, server/worker monitoring with atlas, and
genie job monitoring.
On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
<ge...@cloverhealth.com.invalid> wrote:
> It's presumably useful if you want to package your plugins for other people
> to use but it seems like everyone just adds those directly to the Airflow
> codebase these days.
>
> On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <ha...@gmail.com> wrote:
>
> > Yeah so far I have only written hooks and operators so maybe the benefit
> > only kicks in for other airflow abstractions.
> >
> > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > george@cloverhealth.com.INVALID> wrote:
> > >
> > > We also import our operators and sensors directly.
> > >
> > > However, executors and some other pieces are a little bit harder to
> deal
> > > with as non-plugins
> > >
> > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <ha...@gmail.com>
> > wrote:
> > >>
> > >> Hello,
> > >>
> > >> I just got done writing a few plugins, and the process has left me
> > >> wondering what the real benefits are? As far as I can tell, it makes
> > >> testing more difficult since you cannot import from the created
> module,
> > you
> > >> have to import directly from the plugin. Additionally, your code
> editor
> > >> isn't aware of these new plugin modules since they are created when
> you
> > >> start the app up, this makes it seem like there are errors when there
> > >> aren't. Why not just create a lib/ dir with hooks, operators etc..
> dirs
> > >> inside and be done with it? Very curious what peoples thoughts are,
> who
> > >> knows I could be testing wrong or writing the plugins wrong. Thanks in
> > >> advance!
> > >>
> >
>
Re: What are the advantages of plugins, not sure I see any?
Posted by George Leslie-Waksman <ge...@cloverhealth.com.INVALID>.
It's presumably useful if you want to package your plugins for other people
to use but it seems like everyone just adds those directly to the Airflow
codebase these days.
On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <ha...@gmail.com> wrote:
> Yeah so far I have only written hooks and operators so maybe the benefit
> only kicks in for other airflow abstractions.
>
> > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> george@cloverhealth.com.INVALID> wrote:
> >
> > We also import our operators and sensors directly.
> >
> > However, executors and some other pieces are a little bit harder to deal
> > with as non-plugins
> >
> >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <ha...@gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> I just got done writing a few plugins, and the process has left me
> >> wondering what the real benefits are? As far as I can tell, it makes
> >> testing more difficult since you cannot import from the created module,
> you
> >> have to import directly from the plugin. Additionally, your code editor
> >> isn't aware of these new plugin modules since they are created when you
> >> start the app up, this makes it seem like there are errors when there
> >> aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
> >> inside and be done with it? Very curious what peoples thoughts are, who
> >> knows I could be testing wrong or writing the plugins wrong. Thanks in
> >> advance!
> >>
>
Re: What are the advantages of plugins, not sure I see any?
Posted by Kyle Hamlin <ha...@gmail.com>.
Yeah so far I have only written hooks and operators so maybe the benefit only kicks in for other airflow abstractions.
> On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <ge...@cloverhealth.com.INVALID> wrote:
>
> We also import our operators and sensors directly.
>
> However, executors and some other pieces are a little bit harder to deal
> with as non-plugins
>
>> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <ha...@gmail.com> wrote:
>>
>> Hello,
>>
>> I just got done writing a few plugins, and the process has left me
>> wondering what the real benefits are? As far as I can tell, it makes
>> testing more difficult since you cannot import from the created module, you
>> have to import directly from the plugin. Additionally, your code editor
>> isn't aware of these new plugin modules since they are created when you
>> start the app up, this makes it seem like there are errors when there
>> aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
>> inside and be done with it? Very curious what peoples thoughts are, who
>> knows I could be testing wrong or writing the plugins wrong. Thanks in
>> advance!
>>
Re: What are the advantages of plugins, not sure I see any?
Posted by George Leslie-Waksman <ge...@cloverhealth.com.INVALID>.
We also import our operators and sensors directly.
However, executors and some other pieces are a little bit harder to deal
with as non-plugins
On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <ha...@gmail.com> wrote:
> Hello,
>
> I just got done writing a few plugins, and the process has left me
> wondering what the real benefits are? As far as I can tell, it makes
> testing more difficult since you cannot import from the created module, you
> have to import directly from the plugin. Additionally, your code editor
> isn't aware of these new plugin modules since they are created when you
> start the app up, this makes it seem like there are errors when there
> aren't. Why not just create a lib/ dir with hooks, operators etc.. dirs
> inside and be done with it? Very curious what peoples thoughts are, who
> knows I could be testing wrong or writing the plugins wrong. Thanks in
> advance!
>