Re: "Powered By" (working title) section at the official Airflow Website

[Jarek Potiuk <Ja...@polidea.com>]

Hello everyone.

I want to resurrect the thread about adding  "Ecosystem" page to Apache
Airflow Webste. Since we are cleaning the various sources of documentation
(Mainly CWiki) I've added the issue
https://github.com/apache/airflow/issues/10274 ("Add Ecosystem page to
Airflow Website"). I'd love someone picks it up and implements as a PR to
the website (now since deploying the website fully automated so easy to add
to the website).

I invite everyone to comment on the issue when it comes to the content - I
will be happy to moderate it and include some other sources/links
that people find useful.

In case there will be any concerns about whether we should / should not
include some links we can discuss and possibly vote it here.

J.



On Sat, Feb 15, 2020 at 8:53 PM Jarek Potiuk <Ja...@polidea.com>
wrote:

> and I love the Ecosystem name.
>
> On Sat, Feb 15, 2020 at 8:51 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Anyone else with an opinion against ? if not I will! proceed with adding
>> some  of the Powered bys.
>>
>> Another example of great 'powered by' enabled by Airflow (amongst others)
>> is for example 'dbt' - which seems to be very popular recently - Iam
>> watching it closely.
>>
>> it would be great to pay those tools and platforms back their trust in
>> Airflow :) by linking to them.
>>
>> J.
>>
>>
>> On Tue, Jan 21, 2020 at 7:35 PM Aizhamal Nurmamat kyzy <
>> aizhamal@apache.org> wrote:
>>
>>> +1 on having both "Powered by" and  "Ecosystem" sections. Another apache
>>> project example that has these two pages is Flink:
>>> --> https://flink.apache.org/poweredby.html
>>> --> https://flink.apache.org/ecosystem.html
>>>
>>> On Fri, Jan 17, 2020 at 6:44 AM Nathan Maynes <na...@gmail.com>
>>> wrote:
>>>
>>> > Some thoughts on adding an Airflow Ecosystem section to the website, or
>>> > something similarly named.
>>> >
>>> > I like the idea of highlighting how Airflow is being used and tend to
>>> > appreciate a small paragraph about HOW it is used by Company XYZ rather
>>> > than just a list of companies.
>>> >
>>> > I also think linking to companies that provide professional support is
>>> > helpful. This should be done with care as other open source projects
>>> tend
>>> > to lean heavily towards a single company or two which can split the
>>> > community. Cassandra is one example that comes to mind. This particular
>>> > area may need to be handled differently than the others.
>>> >
>>> > Highlighting helpful tools and resources seems like a no brainer to
>>> me. The
>>> > way the Groovy Language (https://groovy-lang.org/learn.html) project
>>> does
>>> > it seems to be a good starting point. They list everything in
>>> "Ecosystem"
>>> > and include other projects that support or work well with Groovy.
>>> Something
>>> > similar would help highlight the growing ecosystem around Airflow.
>>> >
>>> > On Fri, Jan 17, 2020 at 9:19 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
>>> >
>>> > wrote:
>>> >
>>> > > TL;DR; I have a proposal to add "Powered By" section (or similar) to
>>> the
>>> > > official Airflow Website
>>> > >
>>> > > This is the result of a discussion we had with other PMCs regarding
>>> > whether
>>> > > we should have links to external services/tools etc. using Apache
>>> > Airflow.
>>> > > We think devlist is the right place to bring it and all the members
>>> of
>>> > our
>>> > > community should have a chance to share their
>>> > > opinions/concerns/expectations.
>>> > >
>>> > > I believe we should have a page where we describe some other sources
>>> of
>>> > > information and some other notable uses of Airflow that are outside
>>> of
>>> > the
>>> > > "official" source for Apache Airflow. Those are outside of the
>>> community
>>> > > control - but nevertheless, super useful for the users.
>>> > >
>>> > > The "working title" for this page could  be "Others about Airflow",
>>> > > "Powered by Airflow" - I am sure we could come with a good name for
>>> it in
>>> > > the community.
>>> > >
>>> > > I think we could make such a page with a very clear disclaimer that
>>> those
>>> > > are useful sources, but outside of the community control and we
>>> should be
>>> > > free to remove any of such pages from the list in case we see that
>>> there
>>> > > are some violations with the Apache Way.
>>> > >
>>> > > I thought about some general areas on such page (and we might decide
>>> to
>>> > > include only some of those):
>>> > >
>>> > > 1) Other sources of information about Airflow
>>> > > - here https://github.com/jghoman/awesome-apache-airflow is
>>> definitely
>>> > > something that should be on the list - it's volunteering effort of
>>> Jakob
>>> > > but highly regarded among the community, maybe other sources could be
>>> > added
>>> > > here.
>>> > >
>>> > > 2)  Who's Using Airflow:
>>> > > - here we could have a list of companies that are using Airflow. We
>>> > already
>>> > > have it in the README.md but I think it really belongs to the website
>>> > > rather than the README.md (but love to hear what others think about
>>> it).
>>> > >
>>> > > 3) Airflow-as-a-service
>>> > > - here we could add Composer, Astronomer and possibly other
>>> > > "airflow-as-a-service" companies
>>> > >
>>> > > 4) Integrating with Airflow:
>>> > > - here we could add all the workflow solutions that somehow integrate
>>> > with
>>> > > Airflow and use it under the hood or export DAGs to Airflow :
>>> > >     - our "Ooozie-2-Airflow" library
>>> > >     - the CWL -> Airflow (https://github.com/Barski-lab/cwl-airflow)
>>> > those
>>> > > CWL developers who wanted to contribute the code to Airflow,
>>> > >     - Databand.ai -> they build specialized ML python DSL to generate
>>> > > Airflow DAGs and use Airflow as execution engine
>>> > >
>>> > > Those are just examples, but I think they can be very useful to be
>>> > > published for the users. However it should be CRYSTAL CLEAR - those
>>> are
>>> > not
>>> > > products/companies Apache Airflow Community has an impact on and
>>> control
>>> > > over.
>>> > >
>>> > > Let me know what you think, I look forward to hear what others think
>>> > about
>>> > > it.
>>> > >
>>> > > J.
>>> > >
>>> > > --
>>> > >
>>> > > Jarek Potiuk
>>> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>>> > >
>>> > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
>>> > <+48%20660%20796%20129>>
>>> > > [image: Polidea] <https://www.polidea.com/>
>>> > >
>>> >
>>> >
>>> > --
>>> > Nathan Maynes <http://bit.ly/115hXAt>
>>> > @nathanmaynes
>>> >
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>