You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by siddharth gupta <si...@gmail.com> on 2018/10/02 00:37:54 UTC

Looking for collaborators

Hello Folks!

I am working on an Airflow Plugin to auto generate DAG code from the UI and
YAML (already have initial design on how it can work etc). I am looking for
folks interested in collaborating. Please reach out to me if you are
interested :)

You can reach out to me at sid88in@gmail.com

Thanks
Sid

Re: Looking for collaborators

Posted by tb...@gmail.com.

I’ve written some code to generate dags from yaml.. would be interested in collaborating 

> On Oct 1, 2018, at 7:43 PM, 김용휘 <fk...@naver.com> wrote:
> 
> I respects you plan.
> so I would like to join in collaborating.
> 
> What can I do for this?
> 
> 
> -----Original Message-----
> From: "siddharth gupta"<si...@gmail.com>
> To: <de...@airflow.incubator.apache.org>;
> Cc:
> Sent: 2018-10-02 (화) 09:37:54
> Subject: Looking for collaborators
> 
> Hello Folks!
> 
> I am working on an Airflow Plugin to auto generate DAG code from the UI and
> YAML (already have initial design on how it can work etc). I am looking for
> folks interested in collaborating. Please reach out to me if you are
> interested :)
> 
> You can reach out to me at sid88in@gmail.com
> 
> Thanks
> Sid
>

RE: Looking for collaborators

Posted by 김용휘 <fk...@naver.com>.

I respects you plan.
so I would like to join in collaborating.

What can I do for this?

-----Original Message-----
From: "siddharth gupta"<si...@gmail.com>
To: <de...@airflow.incubator.apache.org>;
Cc:
Sent: 2018-10-02 (화) 09:37:54
Subject: Looking for collaborators

Hello Folks!

I am working on an Airflow Plugin to auto generate DAG code from the UI and
YAML (already have initial design on how it can work etc). I am looking for
folks interested in collaborating. Please reach out to me if you are
interested :)

You can reach out to me at sid88in@gmail.com

Thanks
Sid

Re: Looking for collaborators

Posted by Soma S Dhavala <so...@gmail.com>.

Hello Everyone,

We recently worked on an open source Machine Learning Work Bench, (not
released to public yet) where a "github for ML models" is envisioned. It is
fundamentally based on the idea that an ML Application a DAG and is
specified via an YAML file and under the hood it is parsed and mapped to an
airflow DAG.

We wanted this tool to be an open source alternative to the commercial
offerings such as a Amazon SageMaker, Google Cloud ML etc.. AirFlow is seen
as one of the alternatives to the orchestrators. Others being Dask for
example.

I see that the compiling/parsing yaml suggested in this threads fits into
what we already did, and there is huge scope to take that idea forward.
More about the principles and the scope is discussed in my blog here
<https://www.linkedin.com/pulse/eleven-commandments-axiomatic-framework-building-machine-soma-dhavala/?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_recent_activity_details_shares%3BF%2FXPrwiLS0uxVNIqR3Zkrw%3D%3D&licu=urn%3Ali%3Acontrol%3Ad_flagship3_profile_view_base_recent_activity_details_shares-link>.
If there is interest, we can push the project to the public and collaborate.

Btw, I initiated a baby experimental project to see the application of
KanBans as a way capture the nodes in a DAG here. Technically it is
possible to spec an ML App as a Kanban and parse it into an airflow DAG.

thanks,
-soma

That specification needs to support CRUD operations an ML App.

yet, where we used AirFlow as one of the DAG orchestrators.

On Tue, Oct 2, 2018 at 12:10 PM Sai Phanindhra <ph...@gmail.com> wrote:

> Hi Siddarth,
>      Your idea sounds interesting and challenging. I would like to
> contribute to it.
>
> On Tue, 2 Oct 2018 at 06:08, siddharth gupta <si...@gmail.com> wrote:
>
> > Hello Folks!
> >
> > I am working on an Airflow Plugin to auto generate DAG code from the UI
> and
> > YAML (already have initial design on how it can work etc). I am looking
> for
> > folks interested in collaborating. Please reach out to me if you are
> > interested :)
> >
> > You can reach out to me at sid88in@gmail.com
> >
> > Thanks
> > Sid
> >
>
>
> --
> Sai Phanindhra,
> Ph: +91 9043258999
>

Re: Looking for collaborators

Posted by Sai Phanindhra <ph...@gmail.com>.

Hi Siddarth,
     Your idea sounds interesting and challenging. I would like to
contribute to it.

On Tue, 2 Oct 2018 at 06:08, siddharth gupta <si...@gmail.com> wrote:

> Hello Folks!
>
> I am working on an Airflow Plugin to auto generate DAG code from the UI and
> YAML (already have initial design on how it can work etc). I am looking for
> folks interested in collaborating. Please reach out to me if you are
> interested :)
>
> You can reach out to me at sid88in@gmail.com
>
> Thanks
> Sid
>


-- 
Sai Phanindhra,
Ph: +91 9043258999

Re: Looking for collaborators

Posted by Iuliia Volkova <xn...@gmail.com>.

Hi, Sid, I will be glad to join. I have experience of custom DAG generating
from yaml files. We did it for 2 clients. From my point of view, all of
those implementations was not optimal, so I will glad share expirience and
help with development if it's needed.
On Tue, Oct 2, 2018 at 3:38 AM siddharth gupta <si...@gmail.com> wrote:

> Hello Folks!
>
> I am working on an Airflow Plugin to auto generate DAG code from the UI and
> YAML (already have initial design on how it can work etc). I am looking for
> folks interested in collaborating. Please reach out to me if you are
> interested :)
>
> You can reach out to me at sid88in@gmail.com
>
> Thanks
> Sid
>

-- 
_________

С уважением, Юлия Волкова
Тел. : +7 (911) 116-71-82