Public Airflow Dev Env

[David Brownkush <da...@gmail.com>]

Hi All,

There is a high obstacle of entry to start contributing to Airflow that might deter new contributors from actually contributing, and that is the complicated environment setup for running pre-commits and tests as described in the quick start guide (not so quick actually). One would need an Ubuntu machine lying around with pycharm installed and decent cpu & memory to run airflow.

What if there were a public server that aspiring contributors could SSH into, skip all the trouble of setups,  dive straight into the code and start working on their first issues? Would anyone care to donate a free machine?

Just a thought; thanks for reading.
David

Re: Public Airflow Dev Env

[Jarek Potiuk <ja...@potiuk.com>]

Just.  Hit.  The.  "." on airflow (or any other) repo in the browser.

You will see the preview of how fast and nice the CodeSpaces will be when
they are available (this is just editor, not environment yet). The dev
environment is coming for everyone by the end of the year (with full breeze
integration from day 1 I hope).

And by the way "t" is nice as well.

J.

On Sat, Jun 19, 2021 at 1:16 AM Leah Cole <co...@google.com.invalid>
wrote:

> Hey! This is something I've been passively thinking a lot about too and
> additionally I've been thinking about how to best keep a maintainers guide,
> because even as a committer, I'm not as knowledgeable as I'd like to be
> about all kinds of airflow maintenance-y things. I'd love to hear if anyone
> has any examples we can learn from of open source projects that have public
> dev envs, as David was thinking about, and/or those who have a maintainers
> guide that you've found really helpful.
>
> Cheers,
> Leah
>
> On Sun, Jun 13, 2021 at 7:54 AM Janardhan <ja...@gmail.com>
> wrote:
>
>> Hi David,
>>
>> Kindly give me a week to complete the feature (I have abandoned previous
>> work recently they have introduced new features like `sudo`, so I will
>> start afresh now). During the development I will reach out.
>>
>> I will start with the basic setup and provide a PR for your review.
>>
>> Thanks again,
>> Janardhan
>>
>> On Sunday, June 13, 2021, David Brownkush <da...@gmail.com>
>> wrote:
>>
>>> If it has the potential to save hours of setup time per user then
>>> there's a lot of value I think. I'd be willing to test or write docs for
>>> the gitpod feature; let us know the link to your branch!
>>>
>>> On 2021/06/13 11:10:13, Janardhan <ja...@apache.org> wrote:
>>> > I am only a user of Airflow.
>>> >
>>> > Gitpod[1] workspaces work fine, there is a vscode option now. And
>>> vscode
>>> > have a good support for python
>>> > related development. There is a chrome extension available so that the
>>> a
>>> > button (for opening a ready-to-code online
>>> >  workspace) shows up at the GitHub repo page.
>>> >
>>> > Since, the discussion is concerned with beginners to the
>>> > project the gitpod works fine.
>>> >
>>> > I recently attempted to add a PR for gitpod, but I did not know whether
>>> > there is interest in this feature.
>>> >
>>> > If you feel this is worth a shot, I will try to contribute gitpod
>>> support.
>>> > (It involves a .gitpod.yml and which has tasks definition for setup
>>> > commands and open ports).
>>> >
>>> > I think the barrier also applies to airflow website - it has complex
>>> commit
>>> > hooks, ssh for submodules, etc.
>>> >
>>> > PS: I am not related to gitpod.io
>>> >
>>> > [1] gitpod.io
>>> >
>>> > Thank you,
>>> > Janardhan
>>> >
>>> >
>>> >
>>> >
>>> > On Sunday, June 13, 2021, Jarek Potiuk <ja...@potiuk.com> wrote:
>>> >
>>> > > I think we can (and will) do better. Setting up and maintaining such
>>> > > machines is quite an effort and cost (especially from
>>> > > security/isolation point of view, protecting against supply-chain
>>> > > attacks but also against people who try to use such environments for
>>> > > bitcoin mining and similar [1] - and there are many more aspects).
>>> > >
>>> > > Just having a machine without having a fully managed lifecycle and
>>> > > someone to solve problems of people using it on a daily basis is not
>>> > > enough.
>>> > >
>>> > > However the plan is (for a long time) to make Airflow fully
>>> integrated
>>> > > with Codespaces [2] when they become generally available.
>>> > >
>>> > > It has been initially planned for Q3 2020 but due to complexity of
>>> > > making it publicly available (and solving the problems I mentioned
>>> > > above) this has been shifted to Q3 2021 (by a year). It isn't an easy
>>> > > thing to release. But I am quite confident GitHub will do it
>>> > > eventually and we will be fully on-board with it.
>>> > >
>>> > > [1] https://www.infoq.com/news/2021/04/GitHub-actions-cryptomining/
>>> > > [2] https://github.com/features/codespaces
>>> > > [3] https://github.com/github/roadmap/issues/55
>>> > >
>>> > > J.
>>> > >
>>> > > On Sun, Jun 13, 2021 at 11:12 AM David Brownkush
>>> > > <da...@gmail.com> wrote:
>>> > > >
>>> > > > Hi All,
>>> > > >
>>> > > > There is a high obstacle of entry to start contributing to Airflow
>>> that
>>> > > might deter new contributors from actually contributing, and that is
>>> the
>>> > > complicated environment setup for running pre-commits and tests as
>>> > > described in the quick start guide (not so quick actually). One
>>> would need
>>> > > an Ubuntu machine lying around with pycharm installed and decent cpu
>>> &
>>> > > memory to run airflow.
>>> > > >
>>> > > > What if there were a public server that aspiring contributors
>>> could SSH
>>> > > into, skip all the trouble of setups,  dive straight into the code
>>> and
>>> > > start working on their first issues? Would anyone care to donate a
>>> free
>>> > > machine?
>>> > > >
>>> > > > Just a thought; thanks for reading.
>>> > > > David
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > +48 660 796 129 <+48%20660%20796%20129>
>>> > >
>>> >
>>>
>>
>
> --
>
> Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com | +1
> (925) 257-2112
> *I'm working weird hours during this pandemic and am sometimes a bit
> slower to respond to PRs/CLs than normal. Please feel free to send me a
> gentle ping for a status update if my slowness is blocking you and I'll do
> my best to give you an ETA. *
>
>

-- 
+48 660 796 129

Re: Public Airflow Dev Env

[Leah Cole <co...@google.com.INVALID>]

Hey! This is something I've been passively thinking a lot about too and
additionally I've been thinking about how to best keep a maintainers guide,
because even as a committer, I'm not as knowledgeable as I'd like to be
about all kinds of airflow maintenance-y things. I'd love to hear if anyone
has any examples we can learn from of open source projects that have public
dev envs, as David was thinking about, and/or those who have a maintainers
guide that you've found really helpful.

Cheers,
Leah

On Sun, Jun 13, 2021 at 7:54 AM Janardhan <ja...@gmail.com>
wrote:

> Hi David,
>
> Kindly give me a week to complete the feature (I have abandoned previous
> work recently they have introduced new features like `sudo`, so I will
> start afresh now). During the development I will reach out.
>
> I will start with the basic setup and provide a PR for your review.
>
> Thanks again,
> Janardhan
>
> On Sunday, June 13, 2021, David Brownkush <da...@gmail.com>
> wrote:
>
>> If it has the potential to save hours of setup time per user then there's
>> a lot of value I think. I'd be willing to test or write docs for the gitpod
>> feature; let us know the link to your branch!
>>
>> On 2021/06/13 11:10:13, Janardhan <ja...@apache.org> wrote:
>> > I am only a user of Airflow.
>> >
>> > Gitpod[1] workspaces work fine, there is a vscode option now. And vscode
>> > have a good support for python
>> > related development. There is a chrome extension available so that the a
>> > button (for opening a ready-to-code online
>> >  workspace) shows up at the GitHub repo page.
>> >
>> > Since, the discussion is concerned with beginners to the
>> > project the gitpod works fine.
>> >
>> > I recently attempted to add a PR for gitpod, but I did not know whether
>> > there is interest in this feature.
>> >
>> > If you feel this is worth a shot, I will try to contribute gitpod
>> support.
>> > (It involves a .gitpod.yml and which has tasks definition for setup
>> > commands and open ports).
>> >
>> > I think the barrier also applies to airflow website - it has complex
>> commit
>> > hooks, ssh for submodules, etc.
>> >
>> > PS: I am not related to gitpod.io
>> >
>> > [1] gitpod.io
>> >
>> > Thank you,
>> > Janardhan
>> >
>> >
>> >
>> >
>> > On Sunday, June 13, 2021, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > > I think we can (and will) do better. Setting up and maintaining such
>> > > machines is quite an effort and cost (especially from
>> > > security/isolation point of view, protecting against supply-chain
>> > > attacks but also against people who try to use such environments for
>> > > bitcoin mining and similar [1] - and there are many more aspects).
>> > >
>> > > Just having a machine without having a fully managed lifecycle and
>> > > someone to solve problems of people using it on a daily basis is not
>> > > enough.
>> > >
>> > > However the plan is (for a long time) to make Airflow fully integrated
>> > > with Codespaces [2] when they become generally available.
>> > >
>> > > It has been initially planned for Q3 2020 but due to complexity of
>> > > making it publicly available (and solving the problems I mentioned
>> > > above) this has been shifted to Q3 2021 (by a year). It isn't an easy
>> > > thing to release. But I am quite confident GitHub will do it
>> > > eventually and we will be fully on-board with it.
>> > >
>> > > [1] https://www.infoq.com/news/2021/04/GitHub-actions-cryptomining/
>> > > [2] https://github.com/features/codespaces
>> > > [3] https://github.com/github/roadmap/issues/55
>> > >
>> > > J.
>> > >
>> > > On Sun, Jun 13, 2021 at 11:12 AM David Brownkush
>> > > <da...@gmail.com> wrote:
>> > > >
>> > > > Hi All,
>> > > >
>> > > > There is a high obstacle of entry to start contributing to Airflow
>> that
>> > > might deter new contributors from actually contributing, and that is
>> the
>> > > complicated environment setup for running pre-commits and tests as
>> > > described in the quick start guide (not so quick actually). One would
>> need
>> > > an Ubuntu machine lying around with pycharm installed and decent cpu &
>> > > memory to run airflow.
>> > > >
>> > > > What if there were a public server that aspiring contributors could
>> SSH
>> > > into, skip all the trouble of setups,  dive straight into the code and
>> > > start working on their first issues? Would anyone care to donate a
>> free
>> > > machine?
>> > > >
>> > > > Just a thought; thanks for reading.
>> > > > David
>> > >
>> > >
>> > >
>> > > --
>> > > +48 660 796 129 <+48%20660%20796%20129>
>> > >
>> >
>>
>

-- 

Leah Cole (she/her) | Developer Programs Engineer | coleleah@google.com | +1
(925) 257-2112
*I'm working weird hours during this pandemic and am sometimes a bit slower
to respond to PRs/CLs than normal. Please feel free to send me a gentle
ping for a status update if my slowness is blocking you and I'll do my best
to give you an ETA. *

Re: Public Airflow Dev Env

[Janardhan <ja...@gmail.com>]

Hi David,

Kindly give me a week to complete the feature (I have abandoned previous
work recently they have introduced new features like `sudo`, so I will
start afresh now). During the development I will reach out.

I will start with the basic setup and provide a PR for your review.

Thanks again,
Janardhan

On Sunday, June 13, 2021, David Brownkush <da...@gmail.com> wrote:

> If it has the potential to save hours of setup time per user then there's
> a lot of value I think. I'd be willing to test or write docs for the gitpod
> feature; let us know the link to your branch!
>
> On 2021/06/13 11:10:13, Janardhan <ja...@apache.org> wrote:
> > I am only a user of Airflow.
> >
> > Gitpod[1] workspaces work fine, there is a vscode option now. And vscode
> > have a good support for python
> > related development. There is a chrome extension available so that the a
> > button (for opening a ready-to-code online
> >  workspace) shows up at the GitHub repo page.
> >
> > Since, the discussion is concerned with beginners to the
> > project the gitpod works fine.
> >
> > I recently attempted to add a PR for gitpod, but I did not know whether
> > there is interest in this feature.
> >
> > If you feel this is worth a shot, I will try to contribute gitpod
> support.
> > (It involves a .gitpod.yml and which has tasks definition for setup
> > commands and open ports).
> >
> > I think the barrier also applies to airflow website - it has complex
> commit
> > hooks, ssh for submodules, etc.
> >
> > PS: I am not related to gitpod.io
> >
> > [1] gitpod.io
> >
> > Thank you,
> > Janardhan
> >
> >
> >
> >
> > On Sunday, June 13, 2021, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I think we can (and will) do better. Setting up and maintaining such
> > > machines is quite an effort and cost (especially from
> > > security/isolation point of view, protecting against supply-chain
> > > attacks but also against people who try to use such environments for
> > > bitcoin mining and similar [1] - and there are many more aspects).
> > >
> > > Just having a machine without having a fully managed lifecycle and
> > > someone to solve problems of people using it on a daily basis is not
> > > enough.
> > >
> > > However the plan is (for a long time) to make Airflow fully integrated
> > > with Codespaces [2] when they become generally available.
> > >
> > > It has been initially planned for Q3 2020 but due to complexity of
> > > making it publicly available (and solving the problems I mentioned
> > > above) this has been shifted to Q3 2021 (by a year). It isn't an easy
> > > thing to release. But I am quite confident GitHub will do it
> > > eventually and we will be fully on-board with it.
> > >
> > > [1] https://www.infoq.com/news/2021/04/GitHub-actions-cryptomining/
> > > [2] https://github.com/features/codespaces
> > > [3] https://github.com/github/roadmap/issues/55
> > >
> > > J.
> > >
> > > On Sun, Jun 13, 2021 at 11:12 AM David Brownkush
> > > <da...@gmail.com> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > There is a high obstacle of entry to start contributing to Airflow
> that
> > > might deter new contributors from actually contributing, and that is
> the
> > > complicated environment setup for running pre-commits and tests as
> > > described in the quick start guide (not so quick actually). One would
> need
> > > an Ubuntu machine lying around with pycharm installed and decent cpu &
> > > memory to run airflow.
> > > >
> > > > What if there were a public server that aspiring contributors could
> SSH
> > > into, skip all the trouble of setups,  dive straight into the code and
> > > start working on their first issues? Would anyone care to donate a free
> > > machine?
> > > >
> > > > Just a thought; thanks for reading.
> > > > David
> > >
> > >
> > >
> > > --
> > > +48 660 796 129
> > >
> >
>

Re: Public Airflow Dev Env

[David Brownkush <da...@gmail.com>]

If it has the potential to save hours of setup time per user then there's a lot of value I think. I'd be willing to test or write docs for the gitpod feature; let us know the link to your branch!

On 2021/06/13 11:10:13, Janardhan <ja...@apache.org> wrote: 
> I am only a user of Airflow.
> 
> Gitpod[1] workspaces work fine, there is a vscode option now. And vscode
> have a good support for python
> related development. There is a chrome extension available so that the a
> button (for opening a ready-to-code online
>  workspace) shows up at the GitHub repo page.
> 
> Since, the discussion is concerned with beginners to the
> project the gitpod works fine.
> 
> I recently attempted to add a PR for gitpod, but I did not know whether
> there is interest in this feature.
> 
> If you feel this is worth a shot, I will try to contribute gitpod support.
> (It involves a .gitpod.yml and which has tasks definition for setup
> commands and open ports).
> 
> I think the barrier also applies to airflow website - it has complex commit
> hooks, ssh for submodules, etc.
> 
> PS: I am not related to gitpod.io
> 
> [1] gitpod.io
> 
> Thank you,
> Janardhan
> 
> 
> 
> 
> On Sunday, June 13, 2021, Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> > I think we can (and will) do better. Setting up and maintaining such
> > machines is quite an effort and cost (especially from
> > security/isolation point of view, protecting against supply-chain
> > attacks but also against people who try to use such environments for
> > bitcoin mining and similar [1] - and there are many more aspects).
> >
> > Just having a machine without having a fully managed lifecycle and
> > someone to solve problems of people using it on a daily basis is not
> > enough.
> >
> > However the plan is (for a long time) to make Airflow fully integrated
> > with Codespaces [2] when they become generally available.
> >
> > It has been initially planned for Q3 2020 but due to complexity of
> > making it publicly available (and solving the problems I mentioned
> > above) this has been shifted to Q3 2021 (by a year). It isn't an easy
> > thing to release. But I am quite confident GitHub will do it
> > eventually and we will be fully on-board with it.
> >
> > [1] https://www.infoq.com/news/2021/04/GitHub-actions-cryptomining/
> > [2] https://github.com/features/codespaces
> > [3] https://github.com/github/roadmap/issues/55
> >
> > J.
> >
> > On Sun, Jun 13, 2021 at 11:12 AM David Brownkush
> > <da...@gmail.com> wrote:
> > >
> > > Hi All,
> > >
> > > There is a high obstacle of entry to start contributing to Airflow that
> > might deter new contributors from actually contributing, and that is the
> > complicated environment setup for running pre-commits and tests as
> > described in the quick start guide (not so quick actually). One would need
> > an Ubuntu machine lying around with pycharm installed and decent cpu &
> > memory to run airflow.
> > >
> > > What if there were a public server that aspiring contributors could SSH
> > into, skip all the trouble of setups,  dive straight into the code and
> > start working on their first issues? Would anyone care to donate a free
> > machine?
> > >
> > > Just a thought; thanks for reading.
> > > David
> >
> >
> >
> > --
> > +48 660 796 129
> >
> 

Re: Public Airflow Dev Env

[Jarek Potiuk <ja...@potiuk.com>]

>
> If you feel this is worth a shot, I will try to contribute gitpod support. (It involves a .gitpod.yml and which has tasks definition for setup commands and open ports).
>

Absolutely. I'd love to review and try it after you make PR.

> I think the barrier also applies to airflow website - it has complex commit hooks, ssh for submodules, etc.

It could be an easier starting point for GitPod as there are quite a
bit less "moving parts" in airflow-site, but big part of it
(submodules and most pre-commits for one) are pretty much the same as
in the main airflow repo.

J

Re: Public Airflow Dev Env

[Janardhan <ja...@apache.org>]

I am only a user of Airflow.

Gitpod[1] workspaces work fine, there is a vscode option now. And vscode
have a good support for python
related development. There is a chrome extension available so that the a
button (for opening a ready-to-code online
 workspace) shows up at the GitHub repo page.

Since, the discussion is concerned with beginners to the
project the gitpod works fine.

I recently attempted to add a PR for gitpod, but I did not know whether
there is interest in this feature.

If you feel this is worth a shot, I will try to contribute gitpod support.
(It involves a .gitpod.yml and which has tasks definition for setup
commands and open ports).

I think the barrier also applies to airflow website - it has complex commit
hooks, ssh for submodules, etc.

PS: I am not related to gitpod.io

[1] gitpod.io

Thank you,
Janardhan




On Sunday, June 13, 2021, Jarek Potiuk <ja...@potiuk.com> wrote:

> I think we can (and will) do better. Setting up and maintaining such
> machines is quite an effort and cost (especially from
> security/isolation point of view, protecting against supply-chain
> attacks but also against people who try to use such environments for
> bitcoin mining and similar [1] - and there are many more aspects).
>
> Just having a machine without having a fully managed lifecycle and
> someone to solve problems of people using it on a daily basis is not
> enough.
>
> However the plan is (for a long time) to make Airflow fully integrated
> with Codespaces [2] when they become generally available.
>
> It has been initially planned for Q3 2020 but due to complexity of
> making it publicly available (and solving the problems I mentioned
> above) this has been shifted to Q3 2021 (by a year). It isn't an easy
> thing to release. But I am quite confident GitHub will do it
> eventually and we will be fully on-board with it.
>
> [1] https://www.infoq.com/news/2021/04/GitHub-actions-cryptomining/
> [2] https://github.com/features/codespaces
> [3] https://github.com/github/roadmap/issues/55
>
> J.
>
> On Sun, Jun 13, 2021 at 11:12 AM David Brownkush
> <da...@gmail.com> wrote:
> >
> > Hi All,
> >
> > There is a high obstacle of entry to start contributing to Airflow that
> might deter new contributors from actually contributing, and that is the
> complicated environment setup for running pre-commits and tests as
> described in the quick start guide (not so quick actually). One would need
> an Ubuntu machine lying around with pycharm installed and decent cpu &
> memory to run airflow.
> >
> > What if there were a public server that aspiring contributors could SSH
> into, skip all the trouble of setups,  dive straight into the code and
> start working on their first issues? Would anyone care to donate a free
> machine?
> >
> > Just a thought; thanks for reading.
> > David
>
>
>
> --
> +48 660 796 129
>

Re: Public Airflow Dev Env

[Jarek Potiuk <ja...@potiuk.com>]

I think we can (and will) do better. Setting up and maintaining such
machines is quite an effort and cost (especially from
security/isolation point of view, protecting against supply-chain
attacks but also against people who try to use such environments for
bitcoin mining and similar [1] - and there are many more aspects).

Just having a machine without having a fully managed lifecycle and
someone to solve problems of people using it on a daily basis is not
enough.

However the plan is (for a long time) to make Airflow fully integrated
with Codespaces [2] when they become generally available.

It has been initially planned for Q3 2020 but due to complexity of
making it publicly available (and solving the problems I mentioned
above) this has been shifted to Q3 2021 (by a year). It isn't an easy
thing to release. But I am quite confident GitHub will do it
eventually and we will be fully on-board with it.

[1] https://www.infoq.com/news/2021/04/GitHub-actions-cryptomining/
[2] https://github.com/features/codespaces
[3] https://github.com/github/roadmap/issues/55

J.

On Sun, Jun 13, 2021 at 11:12 AM David Brownkush
<da...@gmail.com> wrote:
>
> Hi All,
>
> There is a high obstacle of entry to start contributing to Airflow that might deter new contributors from actually contributing, and that is the complicated environment setup for running pre-commits and tests as described in the quick start guide (not so quick actually). One would need an Ubuntu machine lying around with pycharm installed and decent cpu & memory to run airflow.
>
> What if there were a public server that aspiring contributors could SSH into, skip all the trouble of setups,  dive straight into the code and start working on their first issues? Would anyone care to donate a free machine?
>
> Just a thought; thanks for reading.
> David



-- 
+48 660 796 129