You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Gurer Kiratli <gu...@airbnb.com.INVALID> on 2016/08/04 22:21:06 UTC
Airflow Developers Meeting - 08/03 Notes
Agenda
-
Committers sync-up: progress and plans
-
Max to do a recap since last release
-
Airbnb
-
Anyone else?
-
Cooperation Best Practices
-
Solicit for feedback for Impersonation Design Review
-
Release Schedule, Management
-
Roadmap discussion
Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur Wiedmer,
Gurer Kiratli
Notes
-
Recap for the overall project
-
[Max]
https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
-
Airbnb Recap
-
[Paul] Airbnb is continuing work on features related to migrating
more jobs onto Airflow - DB connection scalability, impersonation,
refreshing web UI, DAG git versioning, task resource isolation
-
Other Recaps
-
[Chris] Improvements to Import functionality. Joy did some work
around CLIs and mark success in collaboration with Bolke.
-
[Jeremiah] Came up with the Merge tool. Focusing on the work around
configuration.
-
Meeting cadence
-
[Max] We can have monthly meetings. Let’s start with monthly we can
change the cadence. We should invite all contributors. We should
specify in
the invite that everyone is welcome. Currently posting this on
the mailing
list and wiki.
-
*Action Item* Gurer to schedule monthly meetings next one at Airbnb.
Airbnb is welcoming folks to come onsite
-
Blog/Documentation
-
[Max, Chris, Jeremiah] Having a blog would be useful. We can
structure our thoughts.
-
[Andrew] Blog should have better language with screenshots and all.
This raises the level of effort bar and this might mean that
nobody really
does it.
-
[Max] Maybe use Medium? We can try it out.
-
[Max] Haven’t looked at the documentation for a long time. I don’t
really know about its quality.
-
[Arthur] Looking at the email list responses, seems like the
documentation is missing certain elements.
-
[Max] Maybe we can interview people who just started and identify
gaps.
-
*Action Item* [Jeremiah] We can also ask the mailing list about this.
He will send this out.
-
Cooperation guidelines
-
[Max, Arthur]
https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
-
[Chris] Let’s link the guidelines to contributor md . This should be
discoverable
-
Release
-
[Chris, Max] Bolke might have branch with High charts and Licensing
cherry picked. We can use this as a dry run. We can test the process.
-
[Max] Big release is probably around September.
-
[Andrew] If we want to have a true practice, we need to be diligently
following Apache guidelines.
-
[Max] We might want to revive the email thread about how we should do
this.
-
[Chris] It would be great for someone who hasn’t done this before to
do this.
-
[Andrew] Building a script for this is smart otherwise the manual
work is painful from experience.
https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release
-
*Action Item* [Max] Max will own this. This will happen by 08/19.
-
[Arthur] For the main release around mid September. We need to
confirm this date.
-
Roadmap
-
[Max] Maybe each company puts their own section? Or should we have a
more cohesive roadmap?
-
*Action Item* Gurer to schedule a roadmap discussion
-
Misc
-
[Max] For executor thinking about sticking with Celery and maybe
having Docker implemented on top of it.
-
[Max] We might want to explore Sphinx for generating documentation
from code.
- [Max] Please review the impersonation design review. This is pretty
simple and reversible.
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Chris Riccomini <cr...@apache.org>.
> Maybe we should keep them for Airflow 1.8.
+1
On Sun, Aug 28, 2016 at 8:37 AM, Alex Van Boxel <al...@vanboxel.be> wrote:
> Hey Maxime, I started to list them all and seems to be a lot of them (one
> depending on the other). I think they had some breaking changes. Maybe we
> should keep them for Airflow 1.8. I'll keep running on master for now.
>
>
>
> On Wed, Aug 24, 2016 at 1:59 AM Jakob Homan <jg...@gmail.com> wrote:
>
>> ASF provides per-projects blog infrastructure:
>> http://www.apache.org/dev/project-blogs
>>
>> -Jakob
>>
>>
>> On 23 August 2016 at 15:07, Maxime Beauchemin
>> <ma...@gmail.com> wrote:
>> > I'm planning on working on a minimal apache release this week. If someone
>> > can point to the commit or PRs I can add them to `branch-1.7.2-apache`
>> and
>> > start moving towards the release.
>> >
>> > Max
>> >
>> > On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel <al...@vanboxel.be>
>> wrote:
>> >
>> >> All stuff is in master (I'm running pre-production on master), if it
>> needs
>> >> to be cherry-picked to a branch I can always test it. I would be grate
>> if
>> >> it's in the next release so I can start blogging about it.
>> >>
>> >> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>> >>
>> >> > (Sorry for the late response, I was on holiday)
>> >> >
>> >> > I think the G* operators just need to be cherry picked. This will
>> make us
>> >> > deviate slightly from the
>> >> > previous release, but makes sure we don’t have to ‘fix’ history
>> >> afterwards.
>> >> >
>> >> > Anyone against this?
>> >> >
>> >> > - B.
>> >> >
>> >> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <al...@vanboxel.be>
>> het
>> >> > volgende geschreven:
>> >> > >
>> >> > > Sorry, Bolke. What needs to be done for Google Cloud
>> operators/hooks?
>> >> > >
>> >> > > If you bring me up-to-speed I can do this. I'm currently working an
>> a
>> >> CI
>> >> > > setup for the Google stuff).
>> >> > >
>> >> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com>
>> >> wrote:
>> >> > >
>> >> > >> I have indeed a branch with cherry picked commits. This branch is
>> >> > >> available in the apache repo. The branch is available as
>> >> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we
>> do
>> >> > >> upgrade the google cloud operators/hooks as part of this release as
>> >> the
>> >> > >> licenses were only added after these changes came in. I prefer this
>> >> > >> approach as we stay away from custom commits as much as possible
>> then.
>> >> > >>
>> >> > >> - Bolke
>> >> > >>
>> >> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
>> >> gurer.kiratli@airbnb.com
>> >> > .INVALID>
>> >> > >> het volgende geschreven:
>> >> > >>>
>> >> > >>> Agenda
>> >> > >>>
>> >> > >>>
>> >> > >>> -
>> >> > >>>
>> >> > >>> Committers sync-up: progress and plans
>> >> > >>> -
>> >> > >>>
>> >> > >>> Max to do a recap since last release
>> >> > >>> -
>> >> > >>>
>> >> > >>> Airbnb
>> >> > >>> -
>> >> > >>>
>> >> > >>> Anyone else?
>> >> > >>> -
>> >> > >>>
>> >> > >>> Cooperation Best Practices
>> >> > >>> -
>> >> > >>>
>> >> > >>> Solicit for feedback for Impersonation Design Review
>> >> > >>> -
>> >> > >>>
>> >> > >>> Release Schedule, Management
>> >> > >>> -
>> >> > >>>
>> >> > >>> Roadmap discussion
>> >> > >>>
>> >> > >>>
>> >> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
>> >> Maxime
>> >> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
>> >> > Wiedmer,
>> >> > >>> Gurer Kiratli
>> >> > >>>
>> >> > >>> Notes
>> >> > >>>
>> >> > >>>
>> >> > >>> -
>> >> > >>>
>> >> > >>> Recap for the overall project
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max]
>> >> > >>>
>> >> > >>
>> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
>> >> > >>> -
>> >> > >>>
>> >> > >>> Airbnb Recap
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Paul] Airbnb is continuing work on features related to
>> migrating
>> >> > >>> more jobs onto Airflow - DB connection scalability,
>> >> impersonation,
>> >> > >>> refreshing web UI, DAG git versioning, task resource isolation
>> >> > >>> -
>> >> > >>>
>> >> > >>> Other Recaps
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Chris] Improvements to Import functionality. Joy did some
>> work
>> >> > >>> around CLIs and mark success in collaboration with Bolke.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
>> >> around
>> >> > >>> configuration.
>> >> > >>> -
>> >> > >>>
>> >> > >>> Meeting cadence
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] We can have monthly meetings. Let’s start with monthly
>> we
>> >> can
>> >> > >>> change the cadence. We should invite all contributors. We
>> should
>> >> > >>> specify in
>> >> > >>> the invite that everyone is welcome. Currently posting this on
>> >> > >>> the mailing
>> >> > >>> list and wiki.
>> >> > >>> -
>> >> > >>>
>> >> > >>> *Action Item* Gurer to schedule monthly meetings next one at
>> >> > >> Airbnb.
>> >> > >>> Airbnb is welcoming folks to come onsite
>> >> > >>> -
>> >> > >>>
>> >> > >>> Blog/Documentation
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
>> >> > >>> structure our thoughts.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Andrew] Blog should have better language with screenshots and
>> >> all.
>> >> > >>> This raises the level of effort bar and this might mean that
>> >> > >>> nobody really
>> >> > >>> does it.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] Maybe use Medium? We can try it out.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] Haven’t looked at the documentation for a long time. I
>> >> don’t
>> >> > >>> really know about its quality.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Arthur] Looking at the email list responses, seems like the
>> >> > >>> documentation is missing certain elements.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] Maybe we can interview people who just started and
>> identify
>> >> > >>> gaps.
>> >> > >>> -
>> >> > >>>
>> >> > >>> *Action Item* [Jeremiah] We can also ask the mailing list
>> about
>> >> > >> this.
>> >> > >>> He will send this out.
>> >> > >>> -
>> >> > >>>
>> >> > >>> Cooperation guidelines
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max, Arthur]
>> >> > >>>
>> >> > >>
>> >> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Chris] Let’s link the guidelines to contributor md . This
>> should
>> >> > be
>> >> > >>> discoverable
>> >> > >>> -
>> >> > >>>
>> >> > >>> Release
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Chris, Max] Bolke might have branch with High charts and
>> >> Licensing
>> >> > >>> cherry picked. We can use this as a dry run. We can test the
>> >> > >> process.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] Big release is probably around September.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Andrew] If we want to have a true practice, we need to be
>> >> > >> diligently
>> >> > >>> following Apache guidelines.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] We might want to revive the email thread about how we
>> >> should
>> >> > >> do
>> >> > >>> this.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Chris] It would be great for someone who hasn’t done this
>> before
>> >> > to
>> >> > >>> do this.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Andrew] Building a script for this is smart otherwise the
>> manual
>> >> > >>> work is painful from experience.
>> >> > >>>
>> >> > >>
>> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
>> >> > >>>
>> >> > >>>
>> >> > >>> https://cwiki.apache.org/confluence/display/JCLOUDS/
>> >> Validate+a+Release
>> >> > >>>
>> >> > >>> -
>> >> > >>>
>> >> > >>> *Action Item* [Max] Max will own this. This will happen by
>> 08/19.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Arthur] For the main release around mid September. We need to
>> >> > >>> confirm this date.
>> >> > >>> -
>> >> > >>>
>> >> > >>> Roadmap
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] Maybe each company puts their own section? Or should we
>> >> have
>> >> > a
>> >> > >>> more cohesive roadmap?
>> >> > >>> -
>> >> > >>>
>> >> > >>> *Action Item* Gurer to schedule a roadmap discussion
>> >> > >>> -
>> >> > >>>
>> >> > >>> Misc
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] For executor thinking about sticking with Celery and
>> maybe
>> >> > >>> having Docker implemented on top of it.
>> >> > >>> -
>> >> > >>>
>> >> > >>> [Max] We might want to explore Sphinx for generating
>> >> documentation
>> >> > >>> from code.
>> >> > >>> - [Max] Please review the impersonation design review. This is
>> >> > >> pretty
>> >> > >>> simple and reversible.
>> >> > >>
>> >> > >>
>> >> >
>> >> >
>> >>
>>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Alex Van Boxel <al...@vanboxel.be>.
Hey Maxime, I started to list them all and seems to be a lot of them (one
depending on the other). I think they had some breaking changes. Maybe we
should keep them for Airflow 1.8. I'll keep running on master for now.
On Wed, Aug 24, 2016 at 1:59 AM Jakob Homan <jg...@gmail.com> wrote:
> ASF provides per-projects blog infrastructure:
> http://www.apache.org/dev/project-blogs
>
> -Jakob
>
>
> On 23 August 2016 at 15:07, Maxime Beauchemin
> <ma...@gmail.com> wrote:
> > I'm planning on working on a minimal apache release this week. If someone
> > can point to the commit or PRs I can add them to `branch-1.7.2-apache`
> and
> > start moving towards the release.
> >
> > Max
> >
> > On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel <al...@vanboxel.be>
> wrote:
> >
> >> All stuff is in master (I'm running pre-production on master), if it
> needs
> >> to be cherry-picked to a branch I can always test it. I would be grate
> if
> >> it's in the next release so I can start blogging about it.
> >>
> >> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> > (Sorry for the late response, I was on holiday)
> >> >
> >> > I think the G* operators just need to be cherry picked. This will
> make us
> >> > deviate slightly from the
> >> > previous release, but makes sure we don’t have to ‘fix’ history
> >> afterwards.
> >> >
> >> > Anyone against this?
> >> >
> >> > - B.
> >> >
> >> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <al...@vanboxel.be>
> het
> >> > volgende geschreven:
> >> > >
> >> > > Sorry, Bolke. What needs to be done for Google Cloud
> operators/hooks?
> >> > >
> >> > > If you bring me up-to-speed I can do this. I'm currently working an
> a
> >> CI
> >> > > setup for the Google stuff).
> >> > >
> >> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com>
> >> wrote:
> >> > >
> >> > >> I have indeed a branch with cherry picked commits. This branch is
> >> > >> available in the apache repo. The branch is available as
> >> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we
> do
> >> > >> upgrade the google cloud operators/hooks as part of this release as
> >> the
> >> > >> licenses were only added after these changes came in. I prefer this
> >> > >> approach as we stay away from custom commits as much as possible
> then.
> >> > >>
> >> > >> - Bolke
> >> > >>
> >> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
> >> gurer.kiratli@airbnb.com
> >> > .INVALID>
> >> > >> het volgende geschreven:
> >> > >>>
> >> > >>> Agenda
> >> > >>>
> >> > >>>
> >> > >>> -
> >> > >>>
> >> > >>> Committers sync-up: progress and plans
> >> > >>> -
> >> > >>>
> >> > >>> Max to do a recap since last release
> >> > >>> -
> >> > >>>
> >> > >>> Airbnb
> >> > >>> -
> >> > >>>
> >> > >>> Anyone else?
> >> > >>> -
> >> > >>>
> >> > >>> Cooperation Best Practices
> >> > >>> -
> >> > >>>
> >> > >>> Solicit for feedback for Impersonation Design Review
> >> > >>> -
> >> > >>>
> >> > >>> Release Schedule, Management
> >> > >>> -
> >> > >>>
> >> > >>> Roadmap discussion
> >> > >>>
> >> > >>>
> >> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
> >> Maxime
> >> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> >> > Wiedmer,
> >> > >>> Gurer Kiratli
> >> > >>>
> >> > >>> Notes
> >> > >>>
> >> > >>>
> >> > >>> -
> >> > >>>
> >> > >>> Recap for the overall project
> >> > >>> -
> >> > >>>
> >> > >>> [Max]
> >> > >>>
> >> > >>
> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> >> > >>> -
> >> > >>>
> >> > >>> Airbnb Recap
> >> > >>> -
> >> > >>>
> >> > >>> [Paul] Airbnb is continuing work on features related to
> migrating
> >> > >>> more jobs onto Airflow - DB connection scalability,
> >> impersonation,
> >> > >>> refreshing web UI, DAG git versioning, task resource isolation
> >> > >>> -
> >> > >>>
> >> > >>> Other Recaps
> >> > >>> -
> >> > >>>
> >> > >>> [Chris] Improvements to Import functionality. Joy did some
> work
> >> > >>> around CLIs and mark success in collaboration with Bolke.
> >> > >>> -
> >> > >>>
> >> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
> >> around
> >> > >>> configuration.
> >> > >>> -
> >> > >>>
> >> > >>> Meeting cadence
> >> > >>> -
> >> > >>>
> >> > >>> [Max] We can have monthly meetings. Let’s start with monthly
> we
> >> can
> >> > >>> change the cadence. We should invite all contributors. We
> should
> >> > >>> specify in
> >> > >>> the invite that everyone is welcome. Currently posting this on
> >> > >>> the mailing
> >> > >>> list and wiki.
> >> > >>> -
> >> > >>>
> >> > >>> *Action Item* Gurer to schedule monthly meetings next one at
> >> > >> Airbnb.
> >> > >>> Airbnb is welcoming folks to come onsite
> >> > >>> -
> >> > >>>
> >> > >>> Blog/Documentation
> >> > >>> -
> >> > >>>
> >> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> >> > >>> structure our thoughts.
> >> > >>> -
> >> > >>>
> >> > >>> [Andrew] Blog should have better language with screenshots and
> >> all.
> >> > >>> This raises the level of effort bar and this might mean that
> >> > >>> nobody really
> >> > >>> does it.
> >> > >>> -
> >> > >>>
> >> > >>> [Max] Maybe use Medium? We can try it out.
> >> > >>> -
> >> > >>>
> >> > >>> [Max] Haven’t looked at the documentation for a long time. I
> >> don’t
> >> > >>> really know about its quality.
> >> > >>> -
> >> > >>>
> >> > >>> [Arthur] Looking at the email list responses, seems like the
> >> > >>> documentation is missing certain elements.
> >> > >>> -
> >> > >>>
> >> > >>> [Max] Maybe we can interview people who just started and
> identify
> >> > >>> gaps.
> >> > >>> -
> >> > >>>
> >> > >>> *Action Item* [Jeremiah] We can also ask the mailing list
> about
> >> > >> this.
> >> > >>> He will send this out.
> >> > >>> -
> >> > >>>
> >> > >>> Cooperation guidelines
> >> > >>> -
> >> > >>>
> >> > >>> [Max, Arthur]
> >> > >>>
> >> > >>
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> >> > >>> -
> >> > >>>
> >> > >>> [Chris] Let’s link the guidelines to contributor md . This
> should
> >> > be
> >> > >>> discoverable
> >> > >>> -
> >> > >>>
> >> > >>> Release
> >> > >>> -
> >> > >>>
> >> > >>> [Chris, Max] Bolke might have branch with High charts and
> >> Licensing
> >> > >>> cherry picked. We can use this as a dry run. We can test the
> >> > >> process.
> >> > >>> -
> >> > >>>
> >> > >>> [Max] Big release is probably around September.
> >> > >>> -
> >> > >>>
> >> > >>> [Andrew] If we want to have a true practice, we need to be
> >> > >> diligently
> >> > >>> following Apache guidelines.
> >> > >>> -
> >> > >>>
> >> > >>> [Max] We might want to revive the email thread about how we
> >> should
> >> > >> do
> >> > >>> this.
> >> > >>> -
> >> > >>>
> >> > >>> [Chris] It would be great for someone who hasn’t done this
> before
> >> > to
> >> > >>> do this.
> >> > >>> -
> >> > >>>
> >> > >>> [Andrew] Building a script for this is smart otherwise the
> manual
> >> > >>> work is painful from experience.
> >> > >>>
> >> > >>
> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
> >> > >>>
> >> > >>>
> >> > >>> https://cwiki.apache.org/confluence/display/JCLOUDS/
> >> Validate+a+Release
> >> > >>>
> >> > >>> -
> >> > >>>
> >> > >>> *Action Item* [Max] Max will own this. This will happen by
> 08/19.
> >> > >>> -
> >> > >>>
> >> > >>> [Arthur] For the main release around mid September. We need to
> >> > >>> confirm this date.
> >> > >>> -
> >> > >>>
> >> > >>> Roadmap
> >> > >>> -
> >> > >>>
> >> > >>> [Max] Maybe each company puts their own section? Or should we
> >> have
> >> > a
> >> > >>> more cohesive roadmap?
> >> > >>> -
> >> > >>>
> >> > >>> *Action Item* Gurer to schedule a roadmap discussion
> >> > >>> -
> >> > >>>
> >> > >>> Misc
> >> > >>> -
> >> > >>>
> >> > >>> [Max] For executor thinking about sticking with Celery and
> maybe
> >> > >>> having Docker implemented on top of it.
> >> > >>> -
> >> > >>>
> >> > >>> [Max] We might want to explore Sphinx for generating
> >> documentation
> >> > >>> from code.
> >> > >>> - [Max] Please review the impersonation design review. This is
> >> > >> pretty
> >> > >>> simple and reversible.
> >> > >>
> >> > >>
> >> >
> >> >
> >>
>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Jakob Homan <jg...@gmail.com>.
ASF provides per-projects blog infrastructure:
http://www.apache.org/dev/project-blogs
-Jakob
On 23 August 2016 at 15:07, Maxime Beauchemin
<ma...@gmail.com> wrote:
> I'm planning on working on a minimal apache release this week. If someone
> can point to the commit or PRs I can add them to `branch-1.7.2-apache` and
> start moving towards the release.
>
> Max
>
> On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel <al...@vanboxel.be> wrote:
>
>> All stuff is in master (I'm running pre-production on master), if it needs
>> to be cherry-picked to a branch I can always test it. I would be grate if
>> it's in the next release so I can start blogging about it.
>>
>> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> > (Sorry for the late response, I was on holiday)
>> >
>> > I think the G* operators just need to be cherry picked. This will make us
>> > deviate slightly from the
>> > previous release, but makes sure we don’t have to ‘fix’ history
>> afterwards.
>> >
>> > Anyone against this?
>> >
>> > - B.
>> >
>> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <al...@vanboxel.be> het
>> > volgende geschreven:
>> > >
>> > > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
>> > >
>> > > If you bring me up-to-speed I can do this. I'm currently working an a
>> CI
>> > > setup for the Google stuff).
>> > >
>> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com>
>> wrote:
>> > >
>> > >> I have indeed a branch with cherry picked commits. This branch is
>> > >> available in the apache repo. The branch is available as
>> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
>> > >> upgrade the google cloud operators/hooks as part of this release as
>> the
>> > >> licenses were only added after these changes came in. I prefer this
>> > >> approach as we stay away from custom commits as much as possible then.
>> > >>
>> > >> - Bolke
>> > >>
>> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
>> gurer.kiratli@airbnb.com
>> > .INVALID>
>> > >> het volgende geschreven:
>> > >>>
>> > >>> Agenda
>> > >>>
>> > >>>
>> > >>> -
>> > >>>
>> > >>> Committers sync-up: progress and plans
>> > >>> -
>> > >>>
>> > >>> Max to do a recap since last release
>> > >>> -
>> > >>>
>> > >>> Airbnb
>> > >>> -
>> > >>>
>> > >>> Anyone else?
>> > >>> -
>> > >>>
>> > >>> Cooperation Best Practices
>> > >>> -
>> > >>>
>> > >>> Solicit for feedback for Impersonation Design Review
>> > >>> -
>> > >>>
>> > >>> Release Schedule, Management
>> > >>> -
>> > >>>
>> > >>> Roadmap discussion
>> > >>>
>> > >>>
>> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
>> Maxime
>> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
>> > Wiedmer,
>> > >>> Gurer Kiratli
>> > >>>
>> > >>> Notes
>> > >>>
>> > >>>
>> > >>> -
>> > >>>
>> > >>> Recap for the overall project
>> > >>> -
>> > >>>
>> > >>> [Max]
>> > >>>
>> > >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
>> > >>> -
>> > >>>
>> > >>> Airbnb Recap
>> > >>> -
>> > >>>
>> > >>> [Paul] Airbnb is continuing work on features related to migrating
>> > >>> more jobs onto Airflow - DB connection scalability,
>> impersonation,
>> > >>> refreshing web UI, DAG git versioning, task resource isolation
>> > >>> -
>> > >>>
>> > >>> Other Recaps
>> > >>> -
>> > >>>
>> > >>> [Chris] Improvements to Import functionality. Joy did some work
>> > >>> around CLIs and mark success in collaboration with Bolke.
>> > >>> -
>> > >>>
>> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
>> around
>> > >>> configuration.
>> > >>> -
>> > >>>
>> > >>> Meeting cadence
>> > >>> -
>> > >>>
>> > >>> [Max] We can have monthly meetings. Let’s start with monthly we
>> can
>> > >>> change the cadence. We should invite all contributors. We should
>> > >>> specify in
>> > >>> the invite that everyone is welcome. Currently posting this on
>> > >>> the mailing
>> > >>> list and wiki.
>> > >>> -
>> > >>>
>> > >>> *Action Item* Gurer to schedule monthly meetings next one at
>> > >> Airbnb.
>> > >>> Airbnb is welcoming folks to come onsite
>> > >>> -
>> > >>>
>> > >>> Blog/Documentation
>> > >>> -
>> > >>>
>> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
>> > >>> structure our thoughts.
>> > >>> -
>> > >>>
>> > >>> [Andrew] Blog should have better language with screenshots and
>> all.
>> > >>> This raises the level of effort bar and this might mean that
>> > >>> nobody really
>> > >>> does it.
>> > >>> -
>> > >>>
>> > >>> [Max] Maybe use Medium? We can try it out.
>> > >>> -
>> > >>>
>> > >>> [Max] Haven’t looked at the documentation for a long time. I
>> don’t
>> > >>> really know about its quality.
>> > >>> -
>> > >>>
>> > >>> [Arthur] Looking at the email list responses, seems like the
>> > >>> documentation is missing certain elements.
>> > >>> -
>> > >>>
>> > >>> [Max] Maybe we can interview people who just started and identify
>> > >>> gaps.
>> > >>> -
>> > >>>
>> > >>> *Action Item* [Jeremiah] We can also ask the mailing list about
>> > >> this.
>> > >>> He will send this out.
>> > >>> -
>> > >>>
>> > >>> Cooperation guidelines
>> > >>> -
>> > >>>
>> > >>> [Max, Arthur]
>> > >>>
>> > >>
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
>> > >>> -
>> > >>>
>> > >>> [Chris] Let’s link the guidelines to contributor md . This should
>> > be
>> > >>> discoverable
>> > >>> -
>> > >>>
>> > >>> Release
>> > >>> -
>> > >>>
>> > >>> [Chris, Max] Bolke might have branch with High charts and
>> Licensing
>> > >>> cherry picked. We can use this as a dry run. We can test the
>> > >> process.
>> > >>> -
>> > >>>
>> > >>> [Max] Big release is probably around September.
>> > >>> -
>> > >>>
>> > >>> [Andrew] If we want to have a true practice, we need to be
>> > >> diligently
>> > >>> following Apache guidelines.
>> > >>> -
>> > >>>
>> > >>> [Max] We might want to revive the email thread about how we
>> should
>> > >> do
>> > >>> this.
>> > >>> -
>> > >>>
>> > >>> [Chris] It would be great for someone who hasn’t done this before
>> > to
>> > >>> do this.
>> > >>> -
>> > >>>
>> > >>> [Andrew] Building a script for this is smart otherwise the manual
>> > >>> work is painful from experience.
>> > >>>
>> > >> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
>> > >>>
>> > >>>
>> > >>> https://cwiki.apache.org/confluence/display/JCLOUDS/
>> Validate+a+Release
>> > >>>
>> > >>> -
>> > >>>
>> > >>> *Action Item* [Max] Max will own this. This will happen by 08/19.
>> > >>> -
>> > >>>
>> > >>> [Arthur] For the main release around mid September. We need to
>> > >>> confirm this date.
>> > >>> -
>> > >>>
>> > >>> Roadmap
>> > >>> -
>> > >>>
>> > >>> [Max] Maybe each company puts their own section? Or should we
>> have
>> > a
>> > >>> more cohesive roadmap?
>> > >>> -
>> > >>>
>> > >>> *Action Item* Gurer to schedule a roadmap discussion
>> > >>> -
>> > >>>
>> > >>> Misc
>> > >>> -
>> > >>>
>> > >>> [Max] For executor thinking about sticking with Celery and maybe
>> > >>> having Docker implemented on top of it.
>> > >>> -
>> > >>>
>> > >>> [Max] We might want to explore Sphinx for generating
>> documentation
>> > >>> from code.
>> > >>> - [Max] Please review the impersonation design review. This is
>> > >> pretty
>> > >>> simple and reversible.
>> > >>
>> > >>
>> >
>> >
>>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Maxime Beauchemin <ma...@gmail.com>.
I'm planning on working on a minimal apache release this week. If someone
can point to the commit or PRs I can add them to `branch-1.7.2-apache` and
start moving towards the release.
Max
On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel <al...@vanboxel.be> wrote:
> All stuff is in master (I'm running pre-production on master), if it needs
> to be cherry-picked to a branch I can always test it. I would be grate if
> it's in the next release so I can start blogging about it.
>
> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > (Sorry for the late response, I was on holiday)
> >
> > I think the G* operators just need to be cherry picked. This will make us
> > deviate slightly from the
> > previous release, but makes sure we don’t have to ‘fix’ history
> afterwards.
> >
> > Anyone against this?
> >
> > - B.
> >
> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <al...@vanboxel.be> het
> > volgende geschreven:
> > >
> > > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
> > >
> > > If you bring me up-to-speed I can do this. I'm currently working an a
> CI
> > > setup for the Google stuff).
> > >
> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> > >
> > >> I have indeed a branch with cherry picked commits. This branch is
> > >> available in the apache repo. The branch is available as
> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> > >> upgrade the google cloud operators/hooks as part of this release as
> the
> > >> licenses were only added after these changes came in. I prefer this
> > >> approach as we stay away from custom commits as much as possible then.
> > >>
> > >> - Bolke
> > >>
> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
> gurer.kiratli@airbnb.com
> > .INVALID>
> > >> het volgende geschreven:
> > >>>
> > >>> Agenda
> > >>>
> > >>>
> > >>> -
> > >>>
> > >>> Committers sync-up: progress and plans
> > >>> -
> > >>>
> > >>> Max to do a recap since last release
> > >>> -
> > >>>
> > >>> Airbnb
> > >>> -
> > >>>
> > >>> Anyone else?
> > >>> -
> > >>>
> > >>> Cooperation Best Practices
> > >>> -
> > >>>
> > >>> Solicit for feedback for Impersonation Design Review
> > >>> -
> > >>>
> > >>> Release Schedule, Management
> > >>> -
> > >>>
> > >>> Roadmap discussion
> > >>>
> > >>>
> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
> Maxime
> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> > Wiedmer,
> > >>> Gurer Kiratli
> > >>>
> > >>> Notes
> > >>>
> > >>>
> > >>> -
> > >>>
> > >>> Recap for the overall project
> > >>> -
> > >>>
> > >>> [Max]
> > >>>
> > >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> > >>> -
> > >>>
> > >>> Airbnb Recap
> > >>> -
> > >>>
> > >>> [Paul] Airbnb is continuing work on features related to migrating
> > >>> more jobs onto Airflow - DB connection scalability,
> impersonation,
> > >>> refreshing web UI, DAG git versioning, task resource isolation
> > >>> -
> > >>>
> > >>> Other Recaps
> > >>> -
> > >>>
> > >>> [Chris] Improvements to Import functionality. Joy did some work
> > >>> around CLIs and mark success in collaboration with Bolke.
> > >>> -
> > >>>
> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
> around
> > >>> configuration.
> > >>> -
> > >>>
> > >>> Meeting cadence
> > >>> -
> > >>>
> > >>> [Max] We can have monthly meetings. Let’s start with monthly we
> can
> > >>> change the cadence. We should invite all contributors. We should
> > >>> specify in
> > >>> the invite that everyone is welcome. Currently posting this on
> > >>> the mailing
> > >>> list and wiki.
> > >>> -
> > >>>
> > >>> *Action Item* Gurer to schedule monthly meetings next one at
> > >> Airbnb.
> > >>> Airbnb is welcoming folks to come onsite
> > >>> -
> > >>>
> > >>> Blog/Documentation
> > >>> -
> > >>>
> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> > >>> structure our thoughts.
> > >>> -
> > >>>
> > >>> [Andrew] Blog should have better language with screenshots and
> all.
> > >>> This raises the level of effort bar and this might mean that
> > >>> nobody really
> > >>> does it.
> > >>> -
> > >>>
> > >>> [Max] Maybe use Medium? We can try it out.
> > >>> -
> > >>>
> > >>> [Max] Haven’t looked at the documentation for a long time. I
> don’t
> > >>> really know about its quality.
> > >>> -
> > >>>
> > >>> [Arthur] Looking at the email list responses, seems like the
> > >>> documentation is missing certain elements.
> > >>> -
> > >>>
> > >>> [Max] Maybe we can interview people who just started and identify
> > >>> gaps.
> > >>> -
> > >>>
> > >>> *Action Item* [Jeremiah] We can also ask the mailing list about
> > >> this.
> > >>> He will send this out.
> > >>> -
> > >>>
> > >>> Cooperation guidelines
> > >>> -
> > >>>
> > >>> [Max, Arthur]
> > >>>
> > >>
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> > >>> -
> > >>>
> > >>> [Chris] Let’s link the guidelines to contributor md . This should
> > be
> > >>> discoverable
> > >>> -
> > >>>
> > >>> Release
> > >>> -
> > >>>
> > >>> [Chris, Max] Bolke might have branch with High charts and
> Licensing
> > >>> cherry picked. We can use this as a dry run. We can test the
> > >> process.
> > >>> -
> > >>>
> > >>> [Max] Big release is probably around September.
> > >>> -
> > >>>
> > >>> [Andrew] If we want to have a true practice, we need to be
> > >> diligently
> > >>> following Apache guidelines.
> > >>> -
> > >>>
> > >>> [Max] We might want to revive the email thread about how we
> should
> > >> do
> > >>> this.
> > >>> -
> > >>>
> > >>> [Chris] It would be great for someone who hasn’t done this before
> > to
> > >>> do this.
> > >>> -
> > >>>
> > >>> [Andrew] Building a script for this is smart otherwise the manual
> > >>> work is painful from experience.
> > >>>
> > >> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
> > >>>
> > >>>
> > >>> https://cwiki.apache.org/confluence/display/JCLOUDS/
> Validate+a+Release
> > >>>
> > >>> -
> > >>>
> > >>> *Action Item* [Max] Max will own this. This will happen by 08/19.
> > >>> -
> > >>>
> > >>> [Arthur] For the main release around mid September. We need to
> > >>> confirm this date.
> > >>> -
> > >>>
> > >>> Roadmap
> > >>> -
> > >>>
> > >>> [Max] Maybe each company puts their own section? Or should we
> have
> > a
> > >>> more cohesive roadmap?
> > >>> -
> > >>>
> > >>> *Action Item* Gurer to schedule a roadmap discussion
> > >>> -
> > >>>
> > >>> Misc
> > >>> -
> > >>>
> > >>> [Max] For executor thinking about sticking with Celery and maybe
> > >>> having Docker implemented on top of it.
> > >>> -
> > >>>
> > >>> [Max] We might want to explore Sphinx for generating
> documentation
> > >>> from code.
> > >>> - [Max] Please review the impersonation design review. This is
> > >> pretty
> > >>> simple and reversible.
> > >>
> > >>
> >
> >
>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Alex Van Boxel <al...@vanboxel.be>.
All stuff is in master (I'm running pre-production on master), if it needs
to be cherry-picked to a branch I can always test it. I would be grate if
it's in the next release so I can start blogging about it.
On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin <bd...@gmail.com> wrote:
> (Sorry for the late response, I was on holiday)
>
> I think the G* operators just need to be cherry picked. This will make us
> deviate slightly from the
> previous release, but makes sure we don’t have to ‘fix’ history afterwards.
>
> Anyone against this?
>
> - B.
>
> > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <al...@vanboxel.be> het
> volgende geschreven:
> >
> > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
> >
> > If you bring me up-to-speed I can do this. I'm currently working an a CI
> > setup for the Google stuff).
> >
> > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com> wrote:
> >
> >> I have indeed a branch with cherry picked commits. This branch is
> >> available in the apache repo. The branch is available as
> >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> >> upgrade the google cloud operators/hooks as part of this release as the
> >> licenses were only added after these changes came in. I prefer this
> >> approach as we stay away from custom commits as much as possible then.
> >>
> >> - Bolke
> >>
> >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <gurer.kiratli@airbnb.com
> .INVALID>
> >> het volgende geschreven:
> >>>
> >>> Agenda
> >>>
> >>>
> >>> -
> >>>
> >>> Committers sync-up: progress and plans
> >>> -
> >>>
> >>> Max to do a recap since last release
> >>> -
> >>>
> >>> Airbnb
> >>> -
> >>>
> >>> Anyone else?
> >>> -
> >>>
> >>> Cooperation Best Practices
> >>> -
> >>>
> >>> Solicit for feedback for Impersonation Design Review
> >>> -
> >>>
> >>> Release Schedule, Management
> >>> -
> >>>
> >>> Roadmap discussion
> >>>
> >>>
> >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
> >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
> Wiedmer,
> >>> Gurer Kiratli
> >>>
> >>> Notes
> >>>
> >>>
> >>> -
> >>>
> >>> Recap for the overall project
> >>> -
> >>>
> >>> [Max]
> >>>
> >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> >>> -
> >>>
> >>> Airbnb Recap
> >>> -
> >>>
> >>> [Paul] Airbnb is continuing work on features related to migrating
> >>> more jobs onto Airflow - DB connection scalability, impersonation,
> >>> refreshing web UI, DAG git versioning, task resource isolation
> >>> -
> >>>
> >>> Other Recaps
> >>> -
> >>>
> >>> [Chris] Improvements to Import functionality. Joy did some work
> >>> around CLIs and mark success in collaboration with Bolke.
> >>> -
> >>>
> >>> [Jeremiah] Came up with the Merge tool. Focusing on the work around
> >>> configuration.
> >>> -
> >>>
> >>> Meeting cadence
> >>> -
> >>>
> >>> [Max] We can have monthly meetings. Let’s start with monthly we can
> >>> change the cadence. We should invite all contributors. We should
> >>> specify in
> >>> the invite that everyone is welcome. Currently posting this on
> >>> the mailing
> >>> list and wiki.
> >>> -
> >>>
> >>> *Action Item* Gurer to schedule monthly meetings next one at
> >> Airbnb.
> >>> Airbnb is welcoming folks to come onsite
> >>> -
> >>>
> >>> Blog/Documentation
> >>> -
> >>>
> >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> >>> structure our thoughts.
> >>> -
> >>>
> >>> [Andrew] Blog should have better language with screenshots and all.
> >>> This raises the level of effort bar and this might mean that
> >>> nobody really
> >>> does it.
> >>> -
> >>>
> >>> [Max] Maybe use Medium? We can try it out.
> >>> -
> >>>
> >>> [Max] Haven’t looked at the documentation for a long time. I don’t
> >>> really know about its quality.
> >>> -
> >>>
> >>> [Arthur] Looking at the email list responses, seems like the
> >>> documentation is missing certain elements.
> >>> -
> >>>
> >>> [Max] Maybe we can interview people who just started and identify
> >>> gaps.
> >>> -
> >>>
> >>> *Action Item* [Jeremiah] We can also ask the mailing list about
> >> this.
> >>> He will send this out.
> >>> -
> >>>
> >>> Cooperation guidelines
> >>> -
> >>>
> >>> [Max, Arthur]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> >>> -
> >>>
> >>> [Chris] Let’s link the guidelines to contributor md . This should
> be
> >>> discoverable
> >>> -
> >>>
> >>> Release
> >>> -
> >>>
> >>> [Chris, Max] Bolke might have branch with High charts and Licensing
> >>> cherry picked. We can use this as a dry run. We can test the
> >> process.
> >>> -
> >>>
> >>> [Max] Big release is probably around September.
> >>> -
> >>>
> >>> [Andrew] If we want to have a true practice, we need to be
> >> diligently
> >>> following Apache guidelines.
> >>> -
> >>>
> >>> [Max] We might want to revive the email thread about how we should
> >> do
> >>> this.
> >>> -
> >>>
> >>> [Chris] It would be great for someone who hasn’t done this before
> to
> >>> do this.
> >>> -
> >>>
> >>> [Andrew] Building a script for this is smart otherwise the manual
> >>> work is painful from experience.
> >>>
> >> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
> >>>
> >>>
> >>> https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release
> >>>
> >>> -
> >>>
> >>> *Action Item* [Max] Max will own this. This will happen by 08/19.
> >>> -
> >>>
> >>> [Arthur] For the main release around mid September. We need to
> >>> confirm this date.
> >>> -
> >>>
> >>> Roadmap
> >>> -
> >>>
> >>> [Max] Maybe each company puts their own section? Or should we have
> a
> >>> more cohesive roadmap?
> >>> -
> >>>
> >>> *Action Item* Gurer to schedule a roadmap discussion
> >>> -
> >>>
> >>> Misc
> >>> -
> >>>
> >>> [Max] For executor thinking about sticking with Celery and maybe
> >>> having Docker implemented on top of it.
> >>> -
> >>>
> >>> [Max] We might want to explore Sphinx for generating documentation
> >>> from code.
> >>> - [Max] Please review the impersonation design review. This is
> >> pretty
> >>> simple and reversible.
> >>
> >>
>
>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Bolke de Bruin <bd...@gmail.com>.
(Sorry for the late response, I was on holiday)
I think the G* operators just need to be cherry picked. This will make us deviate slightly from the
previous release, but makes sure we don’t have to ‘fix’ history afterwards.
Anyone against this?
- B.
> Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel <al...@vanboxel.be> het volgende geschreven:
>
> Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
>
> If you bring me up-to-speed I can do this. I'm currently working an a CI
> setup for the Google stuff).
>
> On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
>> I have indeed a branch with cherry picked commits. This branch is
>> available in the apache repo. The branch is available as
>> “branch-1.7.2-apache”. What is left to do is to decide weather we do
>> upgrade the google cloud operators/hooks as part of this release as the
>> licenses were only added after these changes came in. I prefer this
>> approach as we stay away from custom commits as much as possible then.
>>
>> - Bolke
>>
>>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <gu...@airbnb.com.INVALID>
>> het volgende geschreven:
>>>
>>> Agenda
>>>
>>>
>>> -
>>>
>>> Committers sync-up: progress and plans
>>> -
>>>
>>> Max to do a recap since last release
>>> -
>>>
>>> Airbnb
>>> -
>>>
>>> Anyone else?
>>> -
>>>
>>> Cooperation Best Practices
>>> -
>>>
>>> Solicit for feedback for Impersonation Design Review
>>> -
>>>
>>> Release Schedule, Management
>>> -
>>>
>>> Roadmap discussion
>>>
>>>
>>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
>>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur Wiedmer,
>>> Gurer Kiratli
>>>
>>> Notes
>>>
>>>
>>> -
>>>
>>> Recap for the overall project
>>> -
>>>
>>> [Max]
>>>
>> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
>>> -
>>>
>>> Airbnb Recap
>>> -
>>>
>>> [Paul] Airbnb is continuing work on features related to migrating
>>> more jobs onto Airflow - DB connection scalability, impersonation,
>>> refreshing web UI, DAG git versioning, task resource isolation
>>> -
>>>
>>> Other Recaps
>>> -
>>>
>>> [Chris] Improvements to Import functionality. Joy did some work
>>> around CLIs and mark success in collaboration with Bolke.
>>> -
>>>
>>> [Jeremiah] Came up with the Merge tool. Focusing on the work around
>>> configuration.
>>> -
>>>
>>> Meeting cadence
>>> -
>>>
>>> [Max] We can have monthly meetings. Let’s start with monthly we can
>>> change the cadence. We should invite all contributors. We should
>>> specify in
>>> the invite that everyone is welcome. Currently posting this on
>>> the mailing
>>> list and wiki.
>>> -
>>>
>>> *Action Item* Gurer to schedule monthly meetings next one at
>> Airbnb.
>>> Airbnb is welcoming folks to come onsite
>>> -
>>>
>>> Blog/Documentation
>>> -
>>>
>>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
>>> structure our thoughts.
>>> -
>>>
>>> [Andrew] Blog should have better language with screenshots and all.
>>> This raises the level of effort bar and this might mean that
>>> nobody really
>>> does it.
>>> -
>>>
>>> [Max] Maybe use Medium? We can try it out.
>>> -
>>>
>>> [Max] Haven’t looked at the documentation for a long time. I don’t
>>> really know about its quality.
>>> -
>>>
>>> [Arthur] Looking at the email list responses, seems like the
>>> documentation is missing certain elements.
>>> -
>>>
>>> [Max] Maybe we can interview people who just started and identify
>>> gaps.
>>> -
>>>
>>> *Action Item* [Jeremiah] We can also ask the mailing list about
>> this.
>>> He will send this out.
>>> -
>>>
>>> Cooperation guidelines
>>> -
>>>
>>> [Max, Arthur]
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
>>> -
>>>
>>> [Chris] Let’s link the guidelines to contributor md . This should be
>>> discoverable
>>> -
>>>
>>> Release
>>> -
>>>
>>> [Chris, Max] Bolke might have branch with High charts and Licensing
>>> cherry picked. We can use this as a dry run. We can test the
>> process.
>>> -
>>>
>>> [Max] Big release is probably around September.
>>> -
>>>
>>> [Andrew] If we want to have a true practice, we need to be
>> diligently
>>> following Apache guidelines.
>>> -
>>>
>>> [Max] We might want to revive the email thread about how we should
>> do
>>> this.
>>> -
>>>
>>> [Chris] It would be great for someone who hasn’t done this before to
>>> do this.
>>> -
>>>
>>> [Andrew] Building a script for this is smart otherwise the manual
>>> work is painful from experience.
>>>
>> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release
>>>
>>> -
>>>
>>> *Action Item* [Max] Max will own this. This will happen by 08/19.
>>> -
>>>
>>> [Arthur] For the main release around mid September. We need to
>>> confirm this date.
>>> -
>>>
>>> Roadmap
>>> -
>>>
>>> [Max] Maybe each company puts their own section? Or should we have a
>>> more cohesive roadmap?
>>> -
>>>
>>> *Action Item* Gurer to schedule a roadmap discussion
>>> -
>>>
>>> Misc
>>> -
>>>
>>> [Max] For executor thinking about sticking with Celery and maybe
>>> having Docker implemented on top of it.
>>> -
>>>
>>> [Max] We might want to explore Sphinx for generating documentation
>>> from code.
>>> - [Max] Please review the impersonation design review. This is
>> pretty
>>> simple and reversible.
>>
>>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Alex Van Boxel <al...@vanboxel.be>.
Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
If you bring me up-to-speed I can do this. I'm currently working an a CI
setup for the Google stuff).
On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin <bd...@gmail.com> wrote:
> I have indeed a branch with cherry picked commits. This branch is
> available in the apache repo. The branch is available as
> “branch-1.7.2-apache”. What is left to do is to decide weather we do
> upgrade the google cloud operators/hooks as part of this release as the
> licenses were only added after these changes came in. I prefer this
> approach as we stay away from custom commits as much as possible then.
>
> - Bolke
>
> > Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <gu...@airbnb.com.INVALID>
> het volgende geschreven:
> >
> > Agenda
> >
> >
> > -
> >
> > Committers sync-up: progress and plans
> > -
> >
> > Max to do a recap since last release
> > -
> >
> > Airbnb
> > -
> >
> > Anyone else?
> > -
> >
> > Cooperation Best Practices
> > -
> >
> > Solicit for feedback for Impersonation Design Review
> > -
> >
> > Release Schedule, Management
> > -
> >
> > Roadmap discussion
> >
> >
> > Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
> > Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur Wiedmer,
> > Gurer Kiratli
> >
> > Notes
> >
> >
> > -
> >
> > Recap for the overall project
> > -
> >
> > [Max]
> >
> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> > -
> >
> > Airbnb Recap
> > -
> >
> > [Paul] Airbnb is continuing work on features related to migrating
> > more jobs onto Airflow - DB connection scalability, impersonation,
> > refreshing web UI, DAG git versioning, task resource isolation
> > -
> >
> > Other Recaps
> > -
> >
> > [Chris] Improvements to Import functionality. Joy did some work
> > around CLIs and mark success in collaboration with Bolke.
> > -
> >
> > [Jeremiah] Came up with the Merge tool. Focusing on the work around
> > configuration.
> > -
> >
> > Meeting cadence
> > -
> >
> > [Max] We can have monthly meetings. Let’s start with monthly we can
> > change the cadence. We should invite all contributors. We should
> > specify in
> > the invite that everyone is welcome. Currently posting this on
> > the mailing
> > list and wiki.
> > -
> >
> > *Action Item* Gurer to schedule monthly meetings next one at
> Airbnb.
> > Airbnb is welcoming folks to come onsite
> > -
> >
> > Blog/Documentation
> > -
> >
> > [Max, Chris, Jeremiah] Having a blog would be useful. We can
> > structure our thoughts.
> > -
> >
> > [Andrew] Blog should have better language with screenshots and all.
> > This raises the level of effort bar and this might mean that
> > nobody really
> > does it.
> > -
> >
> > [Max] Maybe use Medium? We can try it out.
> > -
> >
> > [Max] Haven’t looked at the documentation for a long time. I don’t
> > really know about its quality.
> > -
> >
> > [Arthur] Looking at the email list responses, seems like the
> > documentation is missing certain elements.
> > -
> >
> > [Max] Maybe we can interview people who just started and identify
> > gaps.
> > -
> >
> > *Action Item* [Jeremiah] We can also ask the mailing list about
> this.
> > He will send this out.
> > -
> >
> > Cooperation guidelines
> > -
> >
> > [Max, Arthur]
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> > -
> >
> > [Chris] Let’s link the guidelines to contributor md . This should be
> > discoverable
> > -
> >
> > Release
> > -
> >
> > [Chris, Max] Bolke might have branch with High charts and Licensing
> > cherry picked. We can use this as a dry run. We can test the
> process.
> > -
> >
> > [Max] Big release is probably around September.
> > -
> >
> > [Andrew] If we want to have a true practice, we need to be
> diligently
> > following Apache guidelines.
> > -
> >
> > [Max] We might want to revive the email thread about how we should
> do
> > this.
> > -
> >
> > [Chris] It would be great for someone who hasn’t done this before to
> > do this.
> > -
> >
> > [Andrew] Building a script for this is smart otherwise the manual
> > work is painful from experience.
> >
> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
> >
> >
> > https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release
> >
> > -
> >
> > *Action Item* [Max] Max will own this. This will happen by 08/19.
> > -
> >
> > [Arthur] For the main release around mid September. We need to
> > confirm this date.
> > -
> >
> > Roadmap
> > -
> >
> > [Max] Maybe each company puts their own section? Or should we have a
> > more cohesive roadmap?
> > -
> >
> > *Action Item* Gurer to schedule a roadmap discussion
> > -
> >
> > Misc
> > -
> >
> > [Max] For executor thinking about sticking with Celery and maybe
> > having Docker implemented on top of it.
> > -
> >
> > [Max] We might want to explore Sphinx for generating documentation
> > from code.
> > - [Max] Please review the impersonation design review. This is
> pretty
> > simple and reversible.
>
>
Re: Airflow Developers Meeting - 08/03 Notes
Posted by Bolke de Bruin <bd...@gmail.com>.
I have indeed a branch with cherry picked commits. This branch is available in the apache repo. The branch is available as “branch-1.7.2-apache”. What is left to do is to decide weather we do upgrade the google cloud operators/hooks as part of this release as the licenses were only added after these changes came in. I prefer this approach as we stay away from custom commits as much as possible then.
- Bolke
> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <gu...@airbnb.com.INVALID> het volgende geschreven:
>
> Agenda
>
>
> -
>
> Committers sync-up: progress and plans
> -
>
> Max to do a recap since last release
> -
>
> Airbnb
> -
>
> Anyone else?
> -
>
> Cooperation Best Practices
> -
>
> Solicit for feedback for Impersonation Design Review
> -
>
> Release Schedule, Management
> -
>
> Roadmap discussion
>
>
> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?), Maxime
> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur Wiedmer,
> Gurer Kiratli
>
> Notes
>
>
> -
>
> Recap for the overall project
> -
>
> [Max]
> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
> -
>
> Airbnb Recap
> -
>
> [Paul] Airbnb is continuing work on features related to migrating
> more jobs onto Airflow - DB connection scalability, impersonation,
> refreshing web UI, DAG git versioning, task resource isolation
> -
>
> Other Recaps
> -
>
> [Chris] Improvements to Import functionality. Joy did some work
> around CLIs and mark success in collaboration with Bolke.
> -
>
> [Jeremiah] Came up with the Merge tool. Focusing on the work around
> configuration.
> -
>
> Meeting cadence
> -
>
> [Max] We can have monthly meetings. Let’s start with monthly we can
> change the cadence. We should invite all contributors. We should
> specify in
> the invite that everyone is welcome. Currently posting this on
> the mailing
> list and wiki.
> -
>
> *Action Item* Gurer to schedule monthly meetings next one at Airbnb.
> Airbnb is welcoming folks to come onsite
> -
>
> Blog/Documentation
> -
>
> [Max, Chris, Jeremiah] Having a blog would be useful. We can
> structure our thoughts.
> -
>
> [Andrew] Blog should have better language with screenshots and all.
> This raises the level of effort bar and this might mean that
> nobody really
> does it.
> -
>
> [Max] Maybe use Medium? We can try it out.
> -
>
> [Max] Haven’t looked at the documentation for a long time. I don’t
> really know about its quality.
> -
>
> [Arthur] Looking at the email list responses, seems like the
> documentation is missing certain elements.
> -
>
> [Max] Maybe we can interview people who just started and identify
> gaps.
> -
>
> *Action Item* [Jeremiah] We can also ask the mailing list about this.
> He will send this out.
> -
>
> Cooperation guidelines
> -
>
> [Max, Arthur]
> https://cwiki.apache.org/confluence/display/AIRFLOW/Community+Guidelines
> -
>
> [Chris] Let’s link the guidelines to contributor md . This should be
> discoverable
> -
>
> Release
> -
>
> [Chris, Max] Bolke might have branch with High charts and Licensing
> cherry picked. We can use this as a dry run. We can test the process.
> -
>
> [Max] Big release is probably around September.
> -
>
> [Andrew] If we want to have a true practice, we need to be diligently
> following Apache guidelines.
> -
>
> [Max] We might want to revive the email thread about how we should do
> this.
> -
>
> [Chris] It would be great for someone who hasn’t done this before to
> do this.
> -
>
> [Andrew] Building a script for this is smart otherwise the manual
> work is painful from experience.
> https://cwiki.apache.org/confluence/display/JCLOUDS/Releasing+jclouds
>
>
> https://cwiki.apache.org/confluence/display/JCLOUDS/Validate+a+Release
>
> -
>
> *Action Item* [Max] Max will own this. This will happen by 08/19.
> -
>
> [Arthur] For the main release around mid September. We need to
> confirm this date.
> -
>
> Roadmap
> -
>
> [Max] Maybe each company puts their own section? Or should we have a
> more cohesive roadmap?
> -
>
> *Action Item* Gurer to schedule a roadmap discussion
> -
>
> Misc
> -
>
> [Max] For executor thinking about sticking with Celery and maybe
> having Docker implemented on top of it.
> -
>
> [Max] We might want to explore Sphinx for generating documentation
> from code.
> - [Max] Please review the impersonation design review. This is pretty
> simple and reversible.