Re: What are the advantages of plugins, not sure I see any?

[James Meickle <jm...@quantopian.com>]

I have internally been working on a stock market date/time operator plugin
that I am hoping to open source soon. Figuring out the best way to combine
Airflow plugin packaging and Python packaging has been rather unpleasant,
to be honest. I'd love to see this revisited in a future version of Airflow
so that plugins are just a layer on top of standard Python packages.

On Fri, Mar 30, 2018 at 4:53 PM, Taylor Edmiston <te...@gmail.com>
wrote:

> We might be an edge case running Airflow as a platform at Astronomer, but
> we make hooks and operators that are reused across many Airflow instances
> by customers.  (Starting to open source at
> https://github.com/airflow-plugins.)  We've also run a Mesos executor as a
> plugin for similar reasons so that as we fix bugs or add features we can
> reuse it across Airflow installs.
>
> To add one more point — we've tossed around the idea of building more
> tooling around plugins, so you could do something like: $ airflow plugin
> install -U github-plugin then import GitHubHook and...
>
> A package install via PyPI could work as well but we haven't seen anyone
> else distributing Airflow plugins as packages yet.
>
> T
>
> *Taylor Edmiston*
> TEdmiston.com <https://www.tedmiston.com/> | Blog
> <http://blog.tedmiston.com>
> Stack Overflow CV <https://stackoverflow.com/story/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor>
>
>
> On Fri, Mar 30, 2018 at 4:37 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > Yes, it makes most sense to just have a `common/hooks` and
> > `common/operators/` in the repo where your DAGs live and import them at
> > will.
> >
> > Max
> >
> > On Fri, Mar 30, 2018 at 1:30 PM, Kyle Hamlin <ha...@gmail.com>
> wrote:
> >
> > > Thanks for the responses! I think my conclusion was similar, they seem
> > good
> > > for redistribution, but if you're only working with operators and hooks
> > and
> > > aren't sharing that code then it might not make too much sense to use
> > them.
> > >
> > > On Fri, Mar 30, 2018 at 4:23 PM Maxime Beauchemin <
> > > maximebeauchemin@gmail.com> wrote:
> > >
> > > > The original intent was to use plugins as a way to share sets of
> > objects
> > > > and applications build on top of Airflow.
> > > >
> > > > For instance it'd be possible to ship the things listed bellow as
> > Airflow
> > > > plugin:
> > > >
> > > > * "validate-and-schedule my query" UI
> > > > * a set of ML-related hooks and operators that match a specific
> > workflow
> > > > * a Hive metastore-browser UI
> > > > * drag and drop UI
> > > > * ...
> > > >
> > > > As far as I know it's not a super popular feature. Maybe the scope of
> > > > Airflow is already large enough without having all that stuff sitting
> > on
> > > > top of it. :)
> > > >
> > > > As George pointed out it could also allow to accelerate the release
> > > cadence
> > > > of sets of things that are currently inside Airflow. Things like
> Google
> > > > Cloud-related operators and hooks could ship as a plugin on their own
> > > > release schedule.
> > > >
> > > > Max
> > > >
> > > > On Thu, Mar 29, 2018 at 11:07 PM, Alex Tronchin-James 949-412-7220
> > > > <(949)%20412-7220> <
> > > > alex.n.james@gmail.com> wrote:
> > > >
> > > > > At Netflix we've put our plugin inside the DAGs folder and pointed
> > the
> > > > > config to it there so we can both import directly in DAGs AND
> update
> > > the
> > > > > plugin as we go. This makes it easy to test changes to operators
> > needed
> > > > for
> > > > > ongoing DAG development in the same PR.
> > > > >
> > > > > The two plugin features I've used which don't translate to the
> direct
> > > > > import approach are custom macros (we provide some internal
> > libraries)
> > > > and
> > > > > UI menu links, which we use for linking local docs describing our
> > > > > deployment and custom operators, server/worker monitoring with
> atlas,
> > > and
> > > > > genie job monitoring.
> > > > >
> > > > > On Thu, Mar 29, 2018 at 4:56 PM George Leslie-Waksman
> > > > > <ge...@cloverhealth.com.invalid> wrote:
> > > > >
> > > > > > It's presumably useful if you want to package your plugins for
> > other
> > > > > people
> > > > > > to use but it seems like everyone just adds those directly to the
> > > > Airflow
> > > > > > codebase these days.
> > > > > >
> > > > > > On Thu, Mar 29, 2018 at 4:27 PM Kyle Hamlin <hamlin.kn@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Yeah so far I have only written hooks and operators so maybe
> the
> > > > > benefit
> > > > > > > only  kicks in for other airflow abstractions.
> > > > > > >
> > > > > > > > On Mar 29, 2018, at 7:15 PM, George Leslie-Waksman <
> > > > > > > george@cloverhealth.com.INVALID> wrote:
> > > > > > > >
> > > > > > > > We also import our operators and sensors directly.
> > > > > > > >
> > > > > > > > However, executors and some other pieces are a little bit
> > harder
> > > to
> > > > > > deal
> > > > > > > > with as non-plugins
> > > > > > > >
> > > > > > > >> On Thu, Mar 29, 2018 at 3:56 PM Kyle Hamlin <
> > > hamlin.kn@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hello,
> > > > > > > >>
> > > > > > > >> I just got done writing a few plugins, and the process has
> > left
> > > me
> > > > > > > >> wondering what the real benefits are? As far as I can tell,
> it
> > > > makes
> > > > > > > >> testing more difficult since you cannot import from the
> > created
> > > > > > module,
> > > > > > > you
> > > > > > > >> have to import directly from the plugin. Additionally, your
> > code
> > > > > > editor
> > > > > > > >> isn't aware of these new plugin modules since they are
> created
> > > > when
> > > > > > you
> > > > > > > >> start the app up, this makes it seem like there are errors
> > when
> > > > > there
> > > > > > > >> aren't. Why not just create a lib/ dir with hooks, operators
> > > etc..
> > > > > > dirs
> > > > > > > >> inside and be done with it? Very curious what peoples
> thoughts
> > > > are,
> > > > > > who
> > > > > > > >> knows I could be testing wrong or writing the plugins wrong.
> > > > Thanks
> > > > > in
> > > > > > > >> advance!
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>