You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Daniel Imberman <da...@gmail.com> on 2017/07/20 16:54:01 UTC
Airflow + Kubernetes discussion
Hello everyone,
Recently there's been a fair amount of discussion regarding the integration
of airflow with kubernetes. If there is interest I would love to host an
e-meeting to discuss this integration. I can go over the architecture as it
stands right now and would love feedback on improvements/features/design. I
could also attempt to get one or two members of google's kubernetes team to
join to discuss best practices.
I'm currently thinking that next Thursday at 11AM PST over zoom.us, though
if there's strong opinions otherwise I'd be glad to propose other times.
Cheers!
Daniel
Re: Airflow + Kubernetes discussion
Posted by Daniel Imberman <da...@gmail.com>.
Hello all, the meeting will be moved to the following zoom.us:
https://zoom.us/j/529334876
Sorry again for the confusion.
On Thu, Jul 27, 2017 at 11:07 AM Daniel Imberman <da...@gmail.com>
wrote:
> Hi Guys, sorry about the technical difficulties. I will be switching this
> call to zoom.us and send out a link shortly
>
> On Thu, Jul 27, 2017 at 11:07 AM Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
>> I see the same thing, "The call is full"
>>
>> On Thu, Jul 27, 2017 at 11:04 AM, Wilson Lian <ww...@google.com.invalid>
>> wrote:
>>
>> > The call is full :(
>> >
>> > On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman <
>> > daniel.imberman@gmail.com
>> > > wrote:
>> >
>> > > Hello everyone,
>> > >
>> > > The kubernetes discussion is beginning now. I am including a link
>> below,
>> > > but it can also be found in the calendar event itself.
>> > >
>> > > https://hangouts.google.com/hangouts/_/calendar/
>> > >
>> ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
>> > >
>> > > On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger <ma...@lumoslabs.com>
>> > > wrote:
>> > >
>> > > > Awesome, thanks!
>> > > >
>> > > > On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
>> > > > daniel.imberman@gmail.com
>> > > > > wrote:
>> > > >
>> > > > > Hi Marc,
>> > > > >
>> > > > > Sorry for the confusion. The meeting will take place over the
>> google
>> > > > > hangout link on the event (zoom was just an early suggestion
>> before I
>> > > > > realized we could just create the google hangout).
>> > > > >
>> > > > > Best,
>> > > > > Daniel
>> > > > >
>> > > > > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <
>> marc@lumoslabs.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Is this still happening over a Zoom link to be distributed, or
>> on
>> > the
>> > > > > > Google Hangouts link already associated with the invite?
>> > > > > >
>> > > > > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
>> > > > > > daniel.imberman@gmail.com
>> > > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Victor,
>> > > > > > >
>> > > > > > > This implementation is about allowing native kubernetes
>> > deployment
>> > > of
>> > > > > > tasks
>> > > > > > > on airflow. There are existing deployments/helm scripts with
>> all
>> > > the
>> > > > > > > components needed to launch an airflow cluster, but these
>> > clusters
>> > > > > > > basically consist of having a set up static workers run with
>> > celery
>> > > > to
>> > > > > > > communicate between pods.
>> > > > > > >
>> > > > > > > The KubernetesExecutor will dynamically create pods for each
>> task
>> > > > > you're
>> > > > > > > running. Your cluster will be significantly more elastic and
>> it
>> > > won't
>> > > > > > waste
>> > > > > > > resources when jobs aren't running.
>> > > > > > >
>> > > > > > > We have also created a k8s operator which will allow people to
>> > > build
>> > > > > pods
>> > > > > > > by linking to images/settings. This will give users a lot more
>> > > > > > flexibility
>> > > > > > > in creating custom operators that launch single pods or entire
>> > > > clusters
>> > > > > > to
>> > > > > > > complete a task.
>> > > > > > >
>> > > > > > > I think our implementation is similar to what you're
>> proposing,
>> > but
>> > > > if
>> > > > > > > there's any improvements you can see I'd be glad to speak
>> further
>> > > > over
>> > > > > DM
>> > > > > > > or during the meet-up this Thursday :).
>> > > > > > >
>> > > > > > > Here is the proposal for a high level overview and the WIP PR
>> for
>> > > > > > > reference:
>> > > > > > >
>> > > > > > >
>> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
>> > > > > action?pageId=71013666
>> > > > > > > https://github.com/apache/incubator-airflow/pull/2414
>> > > > > > >
>> > > > > > > Cheers,
>> > > > > > >
>> > > > > > > Daniel
>> > > > > > >
>> > > > > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <
>> amitk@wepay.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > I would like to join as well.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Amit Kulkarni
>> > > > > > > > Site Reliability Engineer
>> > > > > > > > Mobile: (716)-352-3270 <(716)%20352-3270>
>> <(716)%20352-3270> <(716)%20352-3270>
>> > > > <(716)%20352-3270>
>> > > > > > > >
>> > > > > > > > Payments partner to the platform economy
>> > > > > > > >
>> > > > > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
>> > > > > > > victor.monteiro@ubee.in>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi everyone!
>> > > > > > > > >
>> > > > > > > > > I'm new here and I've read some of the discussions about
>> K8s
>> > +
>> > > > > > Airflow.
>> > > > > > > > And
>> > > > > > > > > I would be very interested to join this meeting.
>> > > > > > > > >
>> > > > > > > > > But I'm not quiet sure what is this integration. Is it to
>> > make
>> > > it
>> > > > > > easy
>> > > > > > > to
>> > > > > > > > > deploy Airflow to K8S and have scheduler, webserver and
>> other
>> > > > > airflow
>> > > > > > > > > components inside K8S pods or is it to be able to run
>> > > dags/tasks
>> > > > in
>> > > > > > > > > isolated pod environments to allow temporary isolated
>> > > > environments
>> > > > > > that
>> > > > > > > > > will be destroyed as soon as the task instance / dag run
>> is
>> > > over?
>> > > > > > > > >
>> > > > > > > > > At the company I work on here in Brazil, we have deployed
>> > > Airflow
>> > > > > > with
>> > > > > > > > K8S
>> > > > > > > > > and we are thinking in a way to run dags/tasks in
>> temporary
>> > > > > isolated
>> > > > > > > pods
>> > > > > > > > > to allow, mainly, any powerful operator like
>> PythonOperator
>> > and
>> > > > > > > > > BashOperator. Something using this
>> > > > > > > > >
>> > > > > > https://kubernetes.io/docs/concepts/workloads/
>> > > controllers/jobs-run-to-
>> > > > > > > > > completion/#what-is-a-job
>> > > > > > > > > .
>> > > > > > > > >
>> > > > > > > > > Ps: Sorry if this wasn't the right thread to discuss about
>> > ir.
>> > > If
>> > > > > > that
>> > > > > > > is
>> > > > > > > > > true, message me and I'll take this message to the right
>> > place.
>> > > > > > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
>> > > > > > > > maximebeauchemin@gmail.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > +1!
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
>> > > > > <fenglu@google.com.invalid
>> > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Would like to join, please kindly invite me.
>> > > > > > > > > > > Thanks!
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
>> > > > > > > > > > > daniel.imberman@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hello everyone,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Recently there's been a fair amount of discussion
>> > > regarding
>> > > > > the
>> > > > > > > > > > > integration
>> > > > > > > > > > > > of airflow with kubernetes. If there is interest I
>> > would
>> > > > love
>> > > > > > to
>> > > > > > > > host
>> > > > > > > > > > an
>> > > > > > > > > > > > e-meeting to discuss this integration. I can go over
>> > the
>> > > > > > > > architecture
>> > > > > > > > > > as
>> > > > > > > > > > > it
>> > > > > > > > > > > > stands right now and would love feedback on
>> > > > > > > > > > > improvements/features/design. I
>> > > > > > > > > > > > could also attempt to get one or two members of
>> > google's
>> > > > > > > kubernetes
>> > > > > > > > > > team
>> > > > > > > > > > > to
>> > > > > > > > > > > > join to discuss best practices.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm currently thinking that next Thursday at 11AM
>> PST
>> > > over
>> > > > > > > zoom.us
>> > > > > > > > ,
>> > > > > > > > > > > though
>> > > > > > > > > > > > if there's strong opinions otherwise I'd be glad to
>> > > propose
>> > > > > > other
>> > > > > > > > > > times.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Cheers!
>> > > > > > > > > > > >
>> > > > > > > > > > > > Daniel
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Airflow + Kubernetes discussion
Posted by Daniel Imberman <da...@gmail.com>.
Hi Guys, sorry about the technical difficulties. I will be switching this
call to zoom.us and send out a link shortly
On Thu, Jul 27, 2017 at 11:07 AM Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:
> I see the same thing, "The call is full"
>
> On Thu, Jul 27, 2017 at 11:04 AM, Wilson Lian <ww...@google.com.invalid>
> wrote:
>
> > The call is full :(
> >
> > On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman <
> > daniel.imberman@gmail.com
> > > wrote:
> >
> > > Hello everyone,
> > >
> > > The kubernetes discussion is beginning now. I am including a link
> below,
> > > but it can also be found in the calendar event itself.
> > >
> > > https://hangouts.google.com/hangouts/_/calendar/
> > >
> ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
> > >
> > > On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger <ma...@lumoslabs.com>
> > > wrote:
> > >
> > > > Awesome, thanks!
> > > >
> > > > On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
> > > > daniel.imberman@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Marc,
> > > > >
> > > > > Sorry for the confusion. The meeting will take place over the
> google
> > > > > hangout link on the event (zoom was just an early suggestion
> before I
> > > > > realized we could just create the google hangout).
> > > > >
> > > > > Best,
> > > > > Daniel
> > > > >
> > > > > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <
> marc@lumoslabs.com>
> > > > > wrote:
> > > > >
> > > > > > Is this still happening over a Zoom link to be distributed, or on
> > the
> > > > > > Google Hangouts link already associated with the invite?
> > > > > >
> > > > > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > > > > > daniel.imberman@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi Victor,
> > > > > > >
> > > > > > > This implementation is about allowing native kubernetes
> > deployment
> > > of
> > > > > > tasks
> > > > > > > on airflow. There are existing deployments/helm scripts with
> all
> > > the
> > > > > > > components needed to launch an airflow cluster, but these
> > clusters
> > > > > > > basically consist of having a set up static workers run with
> > celery
> > > > to
> > > > > > > communicate between pods.
> > > > > > >
> > > > > > > The KubernetesExecutor will dynamically create pods for each
> task
> > > > > you're
> > > > > > > running. Your cluster will be significantly more elastic and it
> > > won't
> > > > > > waste
> > > > > > > resources when jobs aren't running.
> > > > > > >
> > > > > > > We have also created a k8s operator which will allow people to
> > > build
> > > > > pods
> > > > > > > by linking to images/settings. This will give users a lot more
> > > > > > flexibility
> > > > > > > in creating custom operators that launch single pods or entire
> > > > clusters
> > > > > > to
> > > > > > > complete a task.
> > > > > > >
> > > > > > > I think our implementation is similar to what you're proposing,
> > but
> > > > if
> > > > > > > there's any improvements you can see I'd be glad to speak
> further
> > > > over
> > > > > DM
> > > > > > > or during the meet-up this Thursday :).
> > > > > > >
> > > > > > > Here is the proposal for a high level overview and the WIP PR
> for
> > > > > > > reference:
> > > > > > >
> > > > > > >
> > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > > action?pageId=71013666
> > > > > > > https://github.com/apache/incubator-airflow/pull/2414
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Daniel
> > > > > > >
> > > > > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <amitk@wepay.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > I would like to join as well.
> > > > > > > >
> > > > > > > >
> > > > > > > > Amit Kulkarni
> > > > > > > > Site Reliability Engineer
> > > > > > > > Mobile: (716)-352-3270 <(716)%20352-3270>
> <(716)%20352-3270> <(716)%20352-3270>
> > > > <(716)%20352-3270>
> > > > > > > >
> > > > > > > > Payments partner to the platform economy
> > > > > > > >
> > > > > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > > > > > victor.monteiro@ubee.in>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone!
> > > > > > > > >
> > > > > > > > > I'm new here and I've read some of the discussions about
> K8s
> > +
> > > > > > Airflow.
> > > > > > > > And
> > > > > > > > > I would be very interested to join this meeting.
> > > > > > > > >
> > > > > > > > > But I'm not quiet sure what is this integration. Is it to
> > make
> > > it
> > > > > > easy
> > > > > > > to
> > > > > > > > > deploy Airflow to K8S and have scheduler, webserver and
> other
> > > > > airflow
> > > > > > > > > components inside K8S pods or is it to be able to run
> > > dags/tasks
> > > > in
> > > > > > > > > isolated pod environments to allow temporary isolated
> > > > environments
> > > > > > that
> > > > > > > > > will be destroyed as soon as the task instance / dag run is
> > > over?
> > > > > > > > >
> > > > > > > > > At the company I work on here in Brazil, we have deployed
> > > Airflow
> > > > > > with
> > > > > > > > K8S
> > > > > > > > > and we are thinking in a way to run dags/tasks in temporary
> > > > > isolated
> > > > > > > pods
> > > > > > > > > to allow, mainly, any powerful operator like PythonOperator
> > and
> > > > > > > > > BashOperator. Something using this
> > > > > > > > >
> > > > > > https://kubernetes.io/docs/concepts/workloads/
> > > controllers/jobs-run-to-
> > > > > > > > > completion/#what-is-a-job
> > > > > > > > > .
> > > > > > > > >
> > > > > > > > > Ps: Sorry if this wasn't the right thread to discuss about
> > ir.
> > > If
> > > > > > that
> > > > > > > is
> > > > > > > > > true, message me and I'll take this message to the right
> > place.
> > > > > > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > > > > > > maximebeauchemin@gmail.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1!
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
> > > > > <fenglu@google.com.invalid
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Would like to join, please kindly invite me.
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > Recently there's been a fair amount of discussion
> > > regarding
> > > > > the
> > > > > > > > > > > integration
> > > > > > > > > > > > of airflow with kubernetes. If there is interest I
> > would
> > > > love
> > > > > > to
> > > > > > > > host
> > > > > > > > > > an
> > > > > > > > > > > > e-meeting to discuss this integration. I can go over
> > the
> > > > > > > > architecture
> > > > > > > > > > as
> > > > > > > > > > > it
> > > > > > > > > > > > stands right now and would love feedback on
> > > > > > > > > > > improvements/features/design. I
> > > > > > > > > > > > could also attempt to get one or two members of
> > google's
> > > > > > > kubernetes
> > > > > > > > > > team
> > > > > > > > > > > to
> > > > > > > > > > > > join to discuss best practices.
> > > > > > > > > > > >
> > > > > > > > > > > > I'm currently thinking that next Thursday at 11AM PST
> > > over
> > > > > > > zoom.us
> > > > > > > > ,
> > > > > > > > > > > though
> > > > > > > > > > > > if there's strong opinions otherwise I'd be glad to
> > > propose
> > > > > > other
> > > > > > > > > > times.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers!
> > > > > > > > > > > >
> > > > > > > > > > > > Daniel
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Maxime Beauchemin <ma...@gmail.com>.
I see the same thing, "The call is full"
On Thu, Jul 27, 2017 at 11:04 AM, Wilson Lian <ww...@google.com.invalid>
wrote:
> The call is full :(
>
> On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman <
> daniel.imberman@gmail.com
> > wrote:
>
> > Hello everyone,
> >
> > The kubernetes discussion is beginning now. I am including a link below,
> > but it can also be found in the calendar event itself.
> >
> > https://hangouts.google.com/hangouts/_/calendar/
> > ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
> >
> > On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger <ma...@lumoslabs.com>
> > wrote:
> >
> > > Awesome, thanks!
> > >
> > > On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
> > > daniel.imberman@gmail.com
> > > > wrote:
> > >
> > > > Hi Marc,
> > > >
> > > > Sorry for the confusion. The meeting will take place over the google
> > > > hangout link on the event (zoom was just an early suggestion before I
> > > > realized we could just create the google hangout).
> > > >
> > > > Best,
> > > > Daniel
> > > >
> > > > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <ma...@lumoslabs.com>
> > > > wrote:
> > > >
> > > > > Is this still happening over a Zoom link to be distributed, or on
> the
> > > > > Google Hangouts link already associated with the invite?
> > > > >
> > > > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > > > > daniel.imberman@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi Victor,
> > > > > >
> > > > > > This implementation is about allowing native kubernetes
> deployment
> > of
> > > > > tasks
> > > > > > on airflow. There are existing deployments/helm scripts with all
> > the
> > > > > > components needed to launch an airflow cluster, but these
> clusters
> > > > > > basically consist of having a set up static workers run with
> celery
> > > to
> > > > > > communicate between pods.
> > > > > >
> > > > > > The KubernetesExecutor will dynamically create pods for each task
> > > > you're
> > > > > > running. Your cluster will be significantly more elastic and it
> > won't
> > > > > waste
> > > > > > resources when jobs aren't running.
> > > > > >
> > > > > > We have also created a k8s operator which will allow people to
> > build
> > > > pods
> > > > > > by linking to images/settings. This will give users a lot more
> > > > > flexibility
> > > > > > in creating custom operators that launch single pods or entire
> > > clusters
> > > > > to
> > > > > > complete a task.
> > > > > >
> > > > > > I think our implementation is similar to what you're proposing,
> but
> > > if
> > > > > > there's any improvements you can see I'd be glad to speak further
> > > over
> > > > DM
> > > > > > or during the meet-up this Thursday :).
> > > > > >
> > > > > > Here is the proposal for a high level overview and the WIP PR for
> > > > > > reference:
> > > > > >
> > > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > > action?pageId=71013666
> > > > > > https://github.com/apache/incubator-airflow/pull/2414
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Daniel
> > > > > >
> > > > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com>
> > > wrote:
> > > > > >
> > > > > > > I would like to join as well.
> > > > > > >
> > > > > > >
> > > > > > > Amit Kulkarni
> > > > > > > Site Reliability Engineer
> > > > > > > Mobile: (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> > > <(716)%20352-3270>
> > > > > > >
> > > > > > > Payments partner to the platform economy
> > > > > > >
> > > > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > > > > victor.monteiro@ubee.in>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone!
> > > > > > > >
> > > > > > > > I'm new here and I've read some of the discussions about K8s
> +
> > > > > Airflow.
> > > > > > > And
> > > > > > > > I would be very interested to join this meeting.
> > > > > > > >
> > > > > > > > But I'm not quiet sure what is this integration. Is it to
> make
> > it
> > > > > easy
> > > > > > to
> > > > > > > > deploy Airflow to K8S and have scheduler, webserver and other
> > > > airflow
> > > > > > > > components inside K8S pods or is it to be able to run
> > dags/tasks
> > > in
> > > > > > > > isolated pod environments to allow temporary isolated
> > > environments
> > > > > that
> > > > > > > > will be destroyed as soon as the task instance / dag run is
> > over?
> > > > > > > >
> > > > > > > > At the company I work on here in Brazil, we have deployed
> > Airflow
> > > > > with
> > > > > > > K8S
> > > > > > > > and we are thinking in a way to run dags/tasks in temporary
> > > > isolated
> > > > > > pods
> > > > > > > > to allow, mainly, any powerful operator like PythonOperator
> and
> > > > > > > > BashOperator. Something using this
> > > > > > > >
> > > > > https://kubernetes.io/docs/concepts/workloads/
> > controllers/jobs-run-to-
> > > > > > > > completion/#what-is-a-job
> > > > > > > > .
> > > > > > > >
> > > > > > > > Ps: Sorry if this wasn't the right thread to discuss about
> ir.
> > If
> > > > > that
> > > > > > is
> > > > > > > > true, message me and I'll take this message to the right
> place.
> > > > > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > > > > > maximebeauchemin@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1!
> > > > > > > > >
> > > > > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
> > > > <fenglu@google.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Would like to join, please kindly invite me.
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello everyone,
> > > > > > > > > > >
> > > > > > > > > > > Recently there's been a fair amount of discussion
> > regarding
> > > > the
> > > > > > > > > > integration
> > > > > > > > > > > of airflow with kubernetes. If there is interest I
> would
> > > love
> > > > > to
> > > > > > > host
> > > > > > > > > an
> > > > > > > > > > > e-meeting to discuss this integration. I can go over
> the
> > > > > > > architecture
> > > > > > > > > as
> > > > > > > > > > it
> > > > > > > > > > > stands right now and would love feedback on
> > > > > > > > > > improvements/features/design. I
> > > > > > > > > > > could also attempt to get one or two members of
> google's
> > > > > > kubernetes
> > > > > > > > > team
> > > > > > > > > > to
> > > > > > > > > > > join to discuss best practices.
> > > > > > > > > > >
> > > > > > > > > > > I'm currently thinking that next Thursday at 11AM PST
> > over
> > > > > > zoom.us
> > > > > > > ,
> > > > > > > > > > though
> > > > > > > > > > > if there's strong opinions otherwise I'd be glad to
> > propose
> > > > > other
> > > > > > > > > times.
> > > > > > > > > > >
> > > > > > > > > > > Cheers!
> > > > > > > > > > >
> > > > > > > > > > > Daniel
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Wilson Lian <ww...@google.com.INVALID>.
The call is full :(
On Thu, Jul 27, 2017 at 11:02 AM, Daniel Imberman <daniel.imberman@gmail.com
> wrote:
> Hello everyone,
>
> The kubernetes discussion is beginning now. I am including a link below,
> but it can also be found in the calendar event itself.
>
> https://hangouts.google.com/hangouts/_/calendar/
> ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
>
> On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger <ma...@lumoslabs.com>
> wrote:
>
> > Awesome, thanks!
> >
> > On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
> > daniel.imberman@gmail.com
> > > wrote:
> >
> > > Hi Marc,
> > >
> > > Sorry for the confusion. The meeting will take place over the google
> > > hangout link on the event (zoom was just an early suggestion before I
> > > realized we could just create the google hangout).
> > >
> > > Best,
> > > Daniel
> > >
> > > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <ma...@lumoslabs.com>
> > > wrote:
> > >
> > > > Is this still happening over a Zoom link to be distributed, or on the
> > > > Google Hangouts link already associated with the invite?
> > > >
> > > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > > > daniel.imberman@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Victor,
> > > > >
> > > > > This implementation is about allowing native kubernetes deployment
> of
> > > > tasks
> > > > > on airflow. There are existing deployments/helm scripts with all
> the
> > > > > components needed to launch an airflow cluster, but these clusters
> > > > > basically consist of having a set up static workers run with celery
> > to
> > > > > communicate between pods.
> > > > >
> > > > > The KubernetesExecutor will dynamically create pods for each task
> > > you're
> > > > > running. Your cluster will be significantly more elastic and it
> won't
> > > > waste
> > > > > resources when jobs aren't running.
> > > > >
> > > > > We have also created a k8s operator which will allow people to
> build
> > > pods
> > > > > by linking to images/settings. This will give users a lot more
> > > > flexibility
> > > > > in creating custom operators that launch single pods or entire
> > clusters
> > > > to
> > > > > complete a task.
> > > > >
> > > > > I think our implementation is similar to what you're proposing, but
> > if
> > > > > there's any improvements you can see I'd be glad to speak further
> > over
> > > DM
> > > > > or during the meet-up this Thursday :).
> > > > >
> > > > > Here is the proposal for a high level overview and the WIP PR for
> > > > > reference:
> > > > >
> > > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=71013666
> > > > > https://github.com/apache/incubator-airflow/pull/2414
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Daniel
> > > > >
> > > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com>
> > wrote:
> > > > >
> > > > > > I would like to join as well.
> > > > > >
> > > > > >
> > > > > > Amit Kulkarni
> > > > > > Site Reliability Engineer
> > > > > > Mobile: (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> > <(716)%20352-3270>
> > > > > >
> > > > > > Payments partner to the platform economy
> > > > > >
> > > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > > > victor.monteiro@ubee.in>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone!
> > > > > > >
> > > > > > > I'm new here and I've read some of the discussions about K8s +
> > > > Airflow.
> > > > > > And
> > > > > > > I would be very interested to join this meeting.
> > > > > > >
> > > > > > > But I'm not quiet sure what is this integration. Is it to make
> it
> > > > easy
> > > > > to
> > > > > > > deploy Airflow to K8S and have scheduler, webserver and other
> > > airflow
> > > > > > > components inside K8S pods or is it to be able to run
> dags/tasks
> > in
> > > > > > > isolated pod environments to allow temporary isolated
> > environments
> > > > that
> > > > > > > will be destroyed as soon as the task instance / dag run is
> over?
> > > > > > >
> > > > > > > At the company I work on here in Brazil, we have deployed
> Airflow
> > > > with
> > > > > > K8S
> > > > > > > and we are thinking in a way to run dags/tasks in temporary
> > > isolated
> > > > > pods
> > > > > > > to allow, mainly, any powerful operator like PythonOperator and
> > > > > > > BashOperator. Something using this
> > > > > > >
> > > > https://kubernetes.io/docs/concepts/workloads/
> controllers/jobs-run-to-
> > > > > > > completion/#what-is-a-job
> > > > > > > .
> > > > > > >
> > > > > > > Ps: Sorry if this wasn't the right thread to discuss about ir.
> If
> > > > that
> > > > > is
> > > > > > > true, message me and I'll take this message to the right place.
> > > > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > > > > maximebeauchemin@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1!
> > > > > > > >
> > > > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
> > > <fenglu@google.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Would like to join, please kindly invite me.
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > Recently there's been a fair amount of discussion
> regarding
> > > the
> > > > > > > > > integration
> > > > > > > > > > of airflow with kubernetes. If there is interest I would
> > love
> > > > to
> > > > > > host
> > > > > > > > an
> > > > > > > > > > e-meeting to discuss this integration. I can go over the
> > > > > > architecture
> > > > > > > > as
> > > > > > > > > it
> > > > > > > > > > stands right now and would love feedback on
> > > > > > > > > improvements/features/design. I
> > > > > > > > > > could also attempt to get one or two members of google's
> > > > > kubernetes
> > > > > > > > team
> > > > > > > > > to
> > > > > > > > > > join to discuss best practices.
> > > > > > > > > >
> > > > > > > > > > I'm currently thinking that next Thursday at 11AM PST
> over
> > > > > zoom.us
> > > > > > ,
> > > > > > > > > though
> > > > > > > > > > if there's strong opinions otherwise I'd be glad to
> propose
> > > > other
> > > > > > > > times.
> > > > > > > > > >
> > > > > > > > > > Cheers!
> > > > > > > > > >
> > > > > > > > > > Daniel
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Daniel Imberman <da...@gmail.com>.
Hello everyone,
The kubernetes discussion is beginning now. I am including a link below,
but it can also be found in the calendar event itself.
https://hangouts.google.com/hangouts/_/calendar/ZGFuaWVsLmltYmVybWFuQGdtYWlsLmNvbQ.3jcahq63njusjbnc2l4po6t53a?authuser=0
On Thu, Jul 27, 2017 at 10:44 AM Marc Bollinger <ma...@lumoslabs.com> wrote:
> Awesome, thanks!
>
> On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <
> daniel.imberman@gmail.com
> > wrote:
>
> > Hi Marc,
> >
> > Sorry for the confusion. The meeting will take place over the google
> > hangout link on the event (zoom was just an early suggestion before I
> > realized we could just create the google hangout).
> >
> > Best,
> > Daniel
> >
> > On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <ma...@lumoslabs.com>
> > wrote:
> >
> > > Is this still happening over a Zoom link to be distributed, or on the
> > > Google Hangouts link already associated with the invite?
> > >
> > > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > > daniel.imberman@gmail.com
> > > > wrote:
> > >
> > > > Hi Victor,
> > > >
> > > > This implementation is about allowing native kubernetes deployment of
> > > tasks
> > > > on airflow. There are existing deployments/helm scripts with all the
> > > > components needed to launch an airflow cluster, but these clusters
> > > > basically consist of having a set up static workers run with celery
> to
> > > > communicate between pods.
> > > >
> > > > The KubernetesExecutor will dynamically create pods for each task
> > you're
> > > > running. Your cluster will be significantly more elastic and it won't
> > > waste
> > > > resources when jobs aren't running.
> > > >
> > > > We have also created a k8s operator which will allow people to build
> > pods
> > > > by linking to images/settings. This will give users a lot more
> > > flexibility
> > > > in creating custom operators that launch single pods or entire
> clusters
> > > to
> > > > complete a task.
> > > >
> > > > I think our implementation is similar to what you're proposing, but
> if
> > > > there's any improvements you can see I'd be glad to speak further
> over
> > DM
> > > > or during the meet-up this Thursday :).
> > > >
> > > > Here is the proposal for a high level overview and the WIP PR for
> > > > reference:
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/viewpage.
> > action?pageId=71013666
> > > > https://github.com/apache/incubator-airflow/pull/2414
> > > >
> > > > Cheers,
> > > >
> > > > Daniel
> > > >
> > > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com>
> wrote:
> > > >
> > > > > I would like to join as well.
> > > > >
> > > > >
> > > > > Amit Kulkarni
> > > > > Site Reliability Engineer
> > > > > Mobile: (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> <(716)%20352-3270>
> > > > >
> > > > > Payments partner to the platform economy
> > > > >
> > > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > > victor.monteiro@ubee.in>
> > > > > wrote:
> > > > >
> > > > > > Hi everyone!
> > > > > >
> > > > > > I'm new here and I've read some of the discussions about K8s +
> > > Airflow.
> > > > > And
> > > > > > I would be very interested to join this meeting.
> > > > > >
> > > > > > But I'm not quiet sure what is this integration. Is it to make it
> > > easy
> > > > to
> > > > > > deploy Airflow to K8S and have scheduler, webserver and other
> > airflow
> > > > > > components inside K8S pods or is it to be able to run dags/tasks
> in
> > > > > > isolated pod environments to allow temporary isolated
> environments
> > > that
> > > > > > will be destroyed as soon as the task instance / dag run is over?
> > > > > >
> > > > > > At the company I work on here in Brazil, we have deployed Airflow
> > > with
> > > > > K8S
> > > > > > and we are thinking in a way to run dags/tasks in temporary
> > isolated
> > > > pods
> > > > > > to allow, mainly, any powerful operator like PythonOperator and
> > > > > > BashOperator. Something using this
> > > > > >
> > > https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > > > > > completion/#what-is-a-job
> > > > > > .
> > > > > >
> > > > > > Ps: Sorry if this wasn't the right thread to discuss about ir. If
> > > that
> > > > is
> > > > > > true, message me and I'll take this message to the right place.
> > > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > > > maximebeauchemin@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1!
> > > > > > >
> > > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
> > <fenglu@google.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Would like to join, please kindly invite me.
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > Recently there's been a fair amount of discussion regarding
> > the
> > > > > > > > integration
> > > > > > > > > of airflow with kubernetes. If there is interest I would
> love
> > > to
> > > > > host
> > > > > > > an
> > > > > > > > > e-meeting to discuss this integration. I can go over the
> > > > > architecture
> > > > > > > as
> > > > > > > > it
> > > > > > > > > stands right now and would love feedback on
> > > > > > > > improvements/features/design. I
> > > > > > > > > could also attempt to get one or two members of google's
> > > > kubernetes
> > > > > > > team
> > > > > > > > to
> > > > > > > > > join to discuss best practices.
> > > > > > > > >
> > > > > > > > > I'm currently thinking that next Thursday at 11AM PST over
> > > > zoom.us
> > > > > ,
> > > > > > > > though
> > > > > > > > > if there's strong opinions otherwise I'd be glad to propose
> > > other
> > > > > > > times.
> > > > > > > > >
> > > > > > > > > Cheers!
> > > > > > > > >
> > > > > > > > > Daniel
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Marc Bollinger <ma...@lumoslabs.com>.
Awesome, thanks!
On Thu, Jul 27, 2017 at 10:38 AM, Daniel Imberman <daniel.imberman@gmail.com
> wrote:
> Hi Marc,
>
> Sorry for the confusion. The meeting will take place over the google
> hangout link on the event (zoom was just an early suggestion before I
> realized we could just create the google hangout).
>
> Best,
> Daniel
>
> On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <ma...@lumoslabs.com>
> wrote:
>
> > Is this still happening over a Zoom link to be distributed, or on the
> > Google Hangouts link already associated with the invite?
> >
> > On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> > daniel.imberman@gmail.com
> > > wrote:
> >
> > > Hi Victor,
> > >
> > > This implementation is about allowing native kubernetes deployment of
> > tasks
> > > on airflow. There are existing deployments/helm scripts with all the
> > > components needed to launch an airflow cluster, but these clusters
> > > basically consist of having a set up static workers run with celery to
> > > communicate between pods.
> > >
> > > The KubernetesExecutor will dynamically create pods for each task
> you're
> > > running. Your cluster will be significantly more elastic and it won't
> > waste
> > > resources when jobs aren't running.
> > >
> > > We have also created a k8s operator which will allow people to build
> pods
> > > by linking to images/settings. This will give users a lot more
> > flexibility
> > > in creating custom operators that launch single pods or entire clusters
> > to
> > > complete a task.
> > >
> > > I think our implementation is similar to what you're proposing, but if
> > > there's any improvements you can see I'd be glad to speak further over
> DM
> > > or during the meet-up this Thursday :).
> > >
> > > Here is the proposal for a high level overview and the WIP PR for
> > > reference:
> > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=71013666
> > > https://github.com/apache/incubator-airflow/pull/2414
> > >
> > > Cheers,
> > >
> > > Daniel
> > >
> > > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com> wrote:
> > >
> > > > I would like to join as well.
> > > >
> > > >
> > > > Amit Kulkarni
> > > > Site Reliability Engineer
> > > > Mobile: (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> > > >
> > > > Payments partner to the platform economy
> > > >
> > > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > > victor.monteiro@ubee.in>
> > > > wrote:
> > > >
> > > > > Hi everyone!
> > > > >
> > > > > I'm new here and I've read some of the discussions about K8s +
> > Airflow.
> > > > And
> > > > > I would be very interested to join this meeting.
> > > > >
> > > > > But I'm not quiet sure what is this integration. Is it to make it
> > easy
> > > to
> > > > > deploy Airflow to K8S and have scheduler, webserver and other
> airflow
> > > > > components inside K8S pods or is it to be able to run dags/tasks in
> > > > > isolated pod environments to allow temporary isolated environments
> > that
> > > > > will be destroyed as soon as the task instance / dag run is over?
> > > > >
> > > > > At the company I work on here in Brazil, we have deployed Airflow
> > with
> > > > K8S
> > > > > and we are thinking in a way to run dags/tasks in temporary
> isolated
> > > pods
> > > > > to allow, mainly, any powerful operator like PythonOperator and
> > > > > BashOperator. Something using this
> > > > >
> > https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > > > > completion/#what-is-a-job
> > > > > .
> > > > >
> > > > > Ps: Sorry if this wasn't the right thread to discuss about ir. If
> > that
> > > is
> > > > > true, message me and I'll take this message to the right place.
> > > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > > maximebeauchemin@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1!
> > > > > >
> > > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu
> <fenglu@google.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Would like to join, please kindly invite me.
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > > daniel.imberman@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > Recently there's been a fair amount of discussion regarding
> the
> > > > > > > integration
> > > > > > > > of airflow with kubernetes. If there is interest I would love
> > to
> > > > host
> > > > > > an
> > > > > > > > e-meeting to discuss this integration. I can go over the
> > > > architecture
> > > > > > as
> > > > > > > it
> > > > > > > > stands right now and would love feedback on
> > > > > > > improvements/features/design. I
> > > > > > > > could also attempt to get one or two members of google's
> > > kubernetes
> > > > > > team
> > > > > > > to
> > > > > > > > join to discuss best practices.
> > > > > > > >
> > > > > > > > I'm currently thinking that next Thursday at 11AM PST over
> > > zoom.us
> > > > ,
> > > > > > > though
> > > > > > > > if there's strong opinions otherwise I'd be glad to propose
> > other
> > > > > > times.
> > > > > > > >
> > > > > > > > Cheers!
> > > > > > > >
> > > > > > > > Daniel
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Daniel Imberman <da...@gmail.com>.
Hi Marc,
Sorry for the confusion. The meeting will take place over the google
hangout link on the event (zoom was just an early suggestion before I
realized we could just create the google hangout).
Best,
Daniel
On Thu, Jul 27, 2017 at 10:11 AM Marc Bollinger <ma...@lumoslabs.com> wrote:
> Is this still happening over a Zoom link to be distributed, or on the
> Google Hangouts link already associated with the invite?
>
> On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <
> daniel.imberman@gmail.com
> > wrote:
>
> > Hi Victor,
> >
> > This implementation is about allowing native kubernetes deployment of
> tasks
> > on airflow. There are existing deployments/helm scripts with all the
> > components needed to launch an airflow cluster, but these clusters
> > basically consist of having a set up static workers run with celery to
> > communicate between pods.
> >
> > The KubernetesExecutor will dynamically create pods for each task you're
> > running. Your cluster will be significantly more elastic and it won't
> waste
> > resources when jobs aren't running.
> >
> > We have also created a k8s operator which will allow people to build pods
> > by linking to images/settings. This will give users a lot more
> flexibility
> > in creating custom operators that launch single pods or entire clusters
> to
> > complete a task.
> >
> > I think our implementation is similar to what you're proposing, but if
> > there's any improvements you can see I'd be glad to speak further over DM
> > or during the meet-up this Thursday :).
> >
> > Here is the proposal for a high level overview and the WIP PR for
> > reference:
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71013666
> > https://github.com/apache/incubator-airflow/pull/2414
> >
> > Cheers,
> >
> > Daniel
> >
> > On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com> wrote:
> >
> > > I would like to join as well.
> > >
> > >
> > > Amit Kulkarni
> > > Site Reliability Engineer
> > > Mobile: (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270>
> > >
> > > Payments partner to the platform economy
> > >
> > > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> > victor.monteiro@ubee.in>
> > > wrote:
> > >
> > > > Hi everyone!
> > > >
> > > > I'm new here and I've read some of the discussions about K8s +
> Airflow.
> > > And
> > > > I would be very interested to join this meeting.
> > > >
> > > > But I'm not quiet sure what is this integration. Is it to make it
> easy
> > to
> > > > deploy Airflow to K8S and have scheduler, webserver and other airflow
> > > > components inside K8S pods or is it to be able to run dags/tasks in
> > > > isolated pod environments to allow temporary isolated environments
> that
> > > > will be destroyed as soon as the task instance / dag run is over?
> > > >
> > > > At the company I work on here in Brazil, we have deployed Airflow
> with
> > > K8S
> > > > and we are thinking in a way to run dags/tasks in temporary isolated
> > pods
> > > > to allow, mainly, any powerful operator like PythonOperator and
> > > > BashOperator. Something using this
> > > >
> https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > > > completion/#what-is-a-job
> > > > .
> > > >
> > > > Ps: Sorry if this wasn't the right thread to discuss about ir. If
> that
> > is
> > > > true, message me and I'll take this message to the right place.
> > > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > > maximebeauchemin@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > +1!
> > > > >
> > > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fenglu@google.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Would like to join, please kindly invite me.
> > > > > > Thanks!
> > > > > >
> > > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > > daniel.imberman@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Recently there's been a fair amount of discussion regarding the
> > > > > > integration
> > > > > > > of airflow with kubernetes. If there is interest I would love
> to
> > > host
> > > > > an
> > > > > > > e-meeting to discuss this integration. I can go over the
> > > architecture
> > > > > as
> > > > > > it
> > > > > > > stands right now and would love feedback on
> > > > > > improvements/features/design. I
> > > > > > > could also attempt to get one or two members of google's
> > kubernetes
> > > > > team
> > > > > > to
> > > > > > > join to discuss best practices.
> > > > > > >
> > > > > > > I'm currently thinking that next Thursday at 11AM PST over
> > zoom.us
> > > ,
> > > > > > though
> > > > > > > if there's strong opinions otherwise I'd be glad to propose
> other
> > > > > times.
> > > > > > >
> > > > > > > Cheers!
> > > > > > >
> > > > > > > Daniel
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Marc Bollinger <ma...@lumoslabs.com>.
Is this still happening over a Zoom link to be distributed, or on the
Google Hangouts link already associated with the invite?
On Sun, Jul 23, 2017 at 12:53 PM, Daniel Imberman <daniel.imberman@gmail.com
> wrote:
> Hi Victor,
>
> This implementation is about allowing native kubernetes deployment of tasks
> on airflow. There are existing deployments/helm scripts with all the
> components needed to launch an airflow cluster, but these clusters
> basically consist of having a set up static workers run with celery to
> communicate between pods.
>
> The KubernetesExecutor will dynamically create pods for each task you're
> running. Your cluster will be significantly more elastic and it won't waste
> resources when jobs aren't running.
>
> We have also created a k8s operator which will allow people to build pods
> by linking to images/settings. This will give users a lot more flexibility
> in creating custom operators that launch single pods or entire clusters to
> complete a task.
>
> I think our implementation is similar to what you're proposing, but if
> there's any improvements you can see I'd be glad to speak further over DM
> or during the meet-up this Thursday :).
>
> Here is the proposal for a high level overview and the WIP PR for
> reference:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71013666
> https://github.com/apache/incubator-airflow/pull/2414
>
> Cheers,
>
> Daniel
>
> On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com> wrote:
>
> > I would like to join as well.
> >
> >
> > Amit Kulkarni
> > Site Reliability Engineer
> > Mobile: (716)-352-3270 <(716)%20352-3270>
> >
> > Payments partner to the platform economy
> >
> > On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <
> victor.monteiro@ubee.in>
> > wrote:
> >
> > > Hi everyone!
> > >
> > > I'm new here and I've read some of the discussions about K8s + Airflow.
> > And
> > > I would be very interested to join this meeting.
> > >
> > > But I'm not quiet sure what is this integration. Is it to make it easy
> to
> > > deploy Airflow to K8S and have scheduler, webserver and other airflow
> > > components inside K8S pods or is it to be able to run dags/tasks in
> > > isolated pod environments to allow temporary isolated environments that
> > > will be destroyed as soon as the task instance / dag run is over?
> > >
> > > At the company I work on here in Brazil, we have deployed Airflow with
> > K8S
> > > and we are thinking in a way to run dags/tasks in temporary isolated
> pods
> > > to allow, mainly, any powerful operator like PythonOperator and
> > > BashOperator. Something using this
> > > https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > > completion/#what-is-a-job
> > > .
> > >
> > > Ps: Sorry if this wasn't the right thread to discuss about ir. If that
> is
> > > true, message me and I'll take this message to the right place.
> > > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> > maximebeauchemin@gmail.com
> > > >
> > > wrote:
> > >
> > > > +1!
> > > >
> > > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fe...@google.com.invalid>
> > > > wrote:
> > > >
> > > > > Would like to join, please kindly invite me.
> > > > > Thanks!
> > > > >
> > > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > > daniel.imberman@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Recently there's been a fair amount of discussion regarding the
> > > > > integration
> > > > > > of airflow with kubernetes. If there is interest I would love to
> > host
> > > > an
> > > > > > e-meeting to discuss this integration. I can go over the
> > architecture
> > > > as
> > > > > it
> > > > > > stands right now and would love feedback on
> > > > > improvements/features/design. I
> > > > > > could also attempt to get one or two members of google's
> kubernetes
> > > > team
> > > > > to
> > > > > > join to discuss best practices.
> > > > > >
> > > > > > I'm currently thinking that next Thursday at 11AM PST over
> zoom.us
> > ,
> > > > > though
> > > > > > if there's strong opinions otherwise I'd be glad to propose other
> > > > times.
> > > > > >
> > > > > > Cheers!
> > > > > >
> > > > > > Daniel
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Daniel Imberman <da...@gmail.com>.
Hi Victor,
This implementation is about allowing native kubernetes deployment of tasks
on airflow. There are existing deployments/helm scripts with all the
components needed to launch an airflow cluster, but these clusters
basically consist of having a set up static workers run with celery to
communicate between pods.
The KubernetesExecutor will dynamically create pods for each task you're
running. Your cluster will be significantly more elastic and it won't waste
resources when jobs aren't running.
We have also created a k8s operator which will allow people to build pods
by linking to images/settings. This will give users a lot more flexibility
in creating custom operators that launch single pods or entire clusters to
complete a task.
I think our implementation is similar to what you're proposing, but if
there's any improvements you can see I'd be glad to speak further over DM
or during the meet-up this Thursday :).
Here is the proposal for a high level overview and the WIP PR for reference:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=71013666
https://github.com/apache/incubator-airflow/pull/2414
Cheers,
Daniel
On Sat, Jul 22, 2017 at 2:24 PM Amit Kulkarni <am...@wepay.com> wrote:
> I would like to join as well.
>
>
> Amit Kulkarni
> Site Reliability Engineer
> Mobile: (716)-352-3270 <(716)%20352-3270>
>
> Payments partner to the platform economy
>
> On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <vi...@ubee.in>
> wrote:
>
> > Hi everyone!
> >
> > I'm new here and I've read some of the discussions about K8s + Airflow.
> And
> > I would be very interested to join this meeting.
> >
> > But I'm not quiet sure what is this integration. Is it to make it easy to
> > deploy Airflow to K8S and have scheduler, webserver and other airflow
> > components inside K8S pods or is it to be able to run dags/tasks in
> > isolated pod environments to allow temporary isolated environments that
> > will be destroyed as soon as the task instance / dag run is over?
> >
> > At the company I work on here in Brazil, we have deployed Airflow with
> K8S
> > and we are thinking in a way to run dags/tasks in temporary isolated pods
> > to allow, mainly, any powerful operator like PythonOperator and
> > BashOperator. Something using this
> > https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> > completion/#what-is-a-job
> > .
> >
> > Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
> > true, message me and I'll take this message to the right place.
> > On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <
> maximebeauchemin@gmail.com
> > >
> > wrote:
> >
> > > +1!
> > >
> > > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fe...@google.com.invalid>
> > > wrote:
> > >
> > > > Would like to join, please kindly invite me.
> > > > Thanks!
> > > >
> > > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > > daniel.imberman@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > Recently there's been a fair amount of discussion regarding the
> > > > integration
> > > > > of airflow with kubernetes. If there is interest I would love to
> host
> > > an
> > > > > e-meeting to discuss this integration. I can go over the
> architecture
> > > as
> > > > it
> > > > > stands right now and would love feedback on
> > > > improvements/features/design. I
> > > > > could also attempt to get one or two members of google's kubernetes
> > > team
> > > > to
> > > > > join to discuss best practices.
> > > > >
> > > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us
> ,
> > > > though
> > > > > if there's strong opinions otherwise I'd be glad to propose other
> > > times.
> > > > >
> > > > > Cheers!
> > > > >
> > > > > Daniel
> > > > >
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Amit Kulkarni <am...@wepay.com>.
I would like to join as well.
Amit Kulkarni
Site Reliability Engineer
Mobile: (716)-352-3270
Payments partner to the platform economy
On Sat, Jul 22, 2017 at 9:07 AM, Victor Monteiro <vi...@ubee.in>
wrote:
> Hi everyone!
>
> I'm new here and I've read some of the discussions about K8s + Airflow. And
> I would be very interested to join this meeting.
>
> But I'm not quiet sure what is this integration. Is it to make it easy to
> deploy Airflow to K8S and have scheduler, webserver and other airflow
> components inside K8S pods or is it to be able to run dags/tasks in
> isolated pod environments to allow temporary isolated environments that
> will be destroyed as soon as the task instance / dag run is over?
>
> At the company I work on here in Brazil, we have deployed Airflow with K8S
> and we are thinking in a way to run dags/tasks in temporary isolated pods
> to allow, mainly, any powerful operator like PythonOperator and
> BashOperator. Something using this
> https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-
> completion/#what-is-a-job
> .
>
> Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
> true, message me and I'll take this message to the right place.
> On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <maximebeauchemin@gmail.com
> >
> wrote:
>
> > +1!
> >
> > On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fe...@google.com.invalid>
> > wrote:
> >
> > > Would like to join, please kindly invite me.
> > > Thanks!
> > >
> > > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > daniel.imberman@gmail.com>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > Recently there's been a fair amount of discussion regarding the
> > > integration
> > > > of airflow with kubernetes. If there is interest I would love to host
> > an
> > > > e-meeting to discuss this integration. I can go over the architecture
> > as
> > > it
> > > > stands right now and would love feedback on
> > > improvements/features/design. I
> > > > could also attempt to get one or two members of google's kubernetes
> > team
> > > to
> > > > join to discuss best practices.
> > > >
> > > > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > > though
> > > > if there's strong opinions otherwise I'd be glad to propose other
> > times.
> > > >
> > > > Cheers!
> > > >
> > > > Daniel
> > > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Victor Monteiro <vi...@ubee.in>.
Hi everyone!
I'm new here and I've read some of the discussions about K8s + Airflow. And
I would be very interested to join this meeting.
But I'm not quiet sure what is this integration. Is it to make it easy to
deploy Airflow to K8S and have scheduler, webserver and other airflow
components inside K8S pods or is it to be able to run dags/tasks in
isolated pod environments to allow temporary isolated environments that
will be destroyed as soon as the task instance / dag run is over?
At the company I work on here in Brazil, we have deployed Airflow with K8S
and we are thinking in a way to run dags/tasks in temporary isolated pods
to allow, mainly, any powerful operator like PythonOperator and
BashOperator. Something using this
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#what-is-a-job
.
Ps: Sorry if this wasn't the right thread to discuss about ir. If that is
true, message me and I'll take this message to the right place.
On Fri, 21 Jul 2017 at 01:55 Maxime Beauchemin <ma...@gmail.com>
wrote:
> +1!
>
> On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fe...@google.com.invalid>
> wrote:
>
> > Would like to join, please kindly invite me.
> > Thanks!
> >
> > On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > daniel.imberman@gmail.com>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Recently there's been a fair amount of discussion regarding the
> > integration
> > > of airflow with kubernetes. If there is interest I would love to host
> an
> > > e-meeting to discuss this integration. I can go over the architecture
> as
> > it
> > > stands right now and would love feedback on
> > improvements/features/design. I
> > > could also attempt to get one or two members of google's kubernetes
> team
> > to
> > > join to discuss best practices.
> > >
> > > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > though
> > > if there's strong opinions otherwise I'd be glad to propose other
> times.
> > >
> > > Cheers!
> > >
> > > Daniel
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Maxime Beauchemin <ma...@gmail.com>.
+1!
On Thu, Jul 20, 2017 at 5:53 PM, Feng Lu <fe...@google.com.invalid> wrote:
> Would like to join, please kindly invite me.
> Thanks!
>
> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > Recently there's been a fair amount of discussion regarding the
> integration
> > of airflow with kubernetes. If there is interest I would love to host an
> > e-meeting to discuss this integration. I can go over the architecture as
> it
> > stands right now and would love feedback on
> improvements/features/design. I
> > could also attempt to get one or two members of google's kubernetes team
> to
> > join to discuss best practices.
> >
> > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> though
> > if there's strong opinions otherwise I'd be glad to propose other times.
> >
> > Cheers!
> >
> > Daniel
> >
>
Re: Airflow + Kubernetes discussion
Posted by Feng Lu <fe...@google.com.INVALID>.
Would like to join, please kindly invite me.
Thanks!
On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <da...@gmail.com>
wrote:
> Hello everyone,
>
> Recently there's been a fair amount of discussion regarding the integration
> of airflow with kubernetes. If there is interest I would love to host an
> e-meeting to discuss this integration. I can go over the architecture as it
> stands right now and would love feedback on improvements/features/design. I
> could also attempt to get one or two members of google's kubernetes team to
> join to discuss best practices.
>
> I'm currently thinking that next Thursday at 11AM PST over zoom.us, though
> if there's strong opinions otherwise I'd be glad to propose other times.
>
> Cheers!
>
> Daniel
>
Re: Airflow + Kubernetes discussion
Posted by Wilson Lian <ww...@google.com.INVALID>.
Please send me an invite!
On Thu, Jul 20, 2017 at 5:15 PM, Dan Davydov <dan.davydov@airbnb.com.invalid
> wrote:
> I'm in.
>
> On Thu, Jul 20, 2017 at 4:30 PM, Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> > Glad to hear that people are interested! I've created a google calendar
> > event and messaged everyone in this thread, if anyone else would like to
> > join please let me know!
> >
> > On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > Invite would be nice, I will try to join!
> > >
> > > > On 20 Jul 2017, at 20:36, Gerard Toonstra <gt...@gmail.com>
> wrote:
> > > >
> > > > send me an invite too!
> > > >
> > > > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin <jl...@apache.org>
> > > wrote:
> > > >
> > > >> I'm interested as well.
> > > >>
> > > >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger <ma...@lumoslabs.com>
> > > wrote:
> > > >>
> > > >>> +1 We're in the middle of moving some services to k8s, and have had
> > our
> > > >>> eye on Airflow.
> > > >>>
> > > >>>> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <
> > > sumeet.manit@gmail.com
> > > >>>
> > > >>> wrote:
> > > >>>>
> > > >>>> I would join as well for sure.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Sumit Maheshwari
> > > >>>> cell. 9632202950
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
> > > >> criccomini@apache.org
> > > >>>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> I would definitely be up to joining. We're interested in the K8s
> > work
> > > >>>>> that's going on. That time works for me.
> > > >>>>>
> > > >>>>> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > > >>>>> daniel.imberman@gmail.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Hello everyone,
> > > >>>>>>
> > > >>>>>> Recently there's been a fair amount of discussion regarding the
> > > >>>>> integration
> > > >>>>>> of airflow with kubernetes. If there is interest I would love to
> > > host
> > > >>> an
> > > >>>>>> e-meeting to discuss this integration. I can go over the
> > > architecture
> > > >>> as
> > > >>>>> it
> > > >>>>>> stands right now and would love feedback on
> > > >>>>> improvements/features/design. I
> > > >>>>>> could also attempt to get one or two members of google's
> > kubernetes
> > > >>> team
> > > >>>>> to
> > > >>>>>> join to discuss best practices.
> > > >>>>>>
> > > >>>>>> I'm currently thinking that next Thursday at 11AM PST over
> > zoom.us,
> > > >>>>> though
> > > >>>>>> if there's strong opinions otherwise I'd be glad to propose
> other
> > > >>> times.
> > > >>>>>>
> > > >>>>>> Cheers!
> > > >>>>>>
> > > >>>>>> Daniel
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> > >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Dan Davydov <da...@airbnb.com.INVALID>.
I'm in.
On Thu, Jul 20, 2017 at 4:30 PM, Daniel Imberman <da...@gmail.com>
wrote:
> Glad to hear that people are interested! I've created a google calendar
> event and messaged everyone in this thread, if anyone else would like to
> join please let me know!
>
> On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Invite would be nice, I will try to join!
> >
> > > On 20 Jul 2017, at 20:36, Gerard Toonstra <gt...@gmail.com> wrote:
> > >
> > > send me an invite too!
> > >
> > > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin <jl...@apache.org>
> > wrote:
> > >
> > >> I'm interested as well.
> > >>
> > >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger <ma...@lumoslabs.com>
> > wrote:
> > >>
> > >>> +1 We're in the middle of moving some services to k8s, and have had
> our
> > >>> eye on Airflow.
> > >>>
> > >>>> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <
> > sumeet.manit@gmail.com
> > >>>
> > >>> wrote:
> > >>>>
> > >>>> I would join as well for sure.
> > >>>>
> > >>>> Thanks,
> > >>>> Sumit Maheshwari
> > >>>> cell. 9632202950
> > >>>>
> > >>>>
> > >>>> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
> > >> criccomini@apache.org
> > >>>>
> > >>>> wrote:
> > >>>>
> > >>>>> I would definitely be up to joining. We're interested in the K8s
> work
> > >>>>> that's going on. That time works for me.
> > >>>>>
> > >>>>> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > >>>>> daniel.imberman@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hello everyone,
> > >>>>>>
> > >>>>>> Recently there's been a fair amount of discussion regarding the
> > >>>>> integration
> > >>>>>> of airflow with kubernetes. If there is interest I would love to
> > host
> > >>> an
> > >>>>>> e-meeting to discuss this integration. I can go over the
> > architecture
> > >>> as
> > >>>>> it
> > >>>>>> stands right now and would love feedback on
> > >>>>> improvements/features/design. I
> > >>>>>> could also attempt to get one or two members of google's
> kubernetes
> > >>> team
> > >>>>> to
> > >>>>>> join to discuss best practices.
> > >>>>>>
> > >>>>>> I'm currently thinking that next Thursday at 11AM PST over
> zoom.us,
> > >>>>> though
> > >>>>>> if there's strong opinions otherwise I'd be glad to propose other
> > >>> times.
> > >>>>>>
> > >>>>>> Cheers!
> > >>>>>>
> > >>>>>> Daniel
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> >
>
Re: Airflow + Kubernetes discussion
Posted by Daniel Imberman <da...@gmail.com>.
Glad to hear that people are interested! I've created a google calendar
event and messaged everyone in this thread, if anyone else would like to
join please let me know!
On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin <bd...@gmail.com> wrote:
> Invite would be nice, I will try to join!
>
> > On 20 Jul 2017, at 20:36, Gerard Toonstra <gt...@gmail.com> wrote:
> >
> > send me an invite too!
> >
> > On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin <jl...@apache.org>
> wrote:
> >
> >> I'm interested as well.
> >>
> >> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger <ma...@lumoslabs.com>
> wrote:
> >>
> >>> +1 We're in the middle of moving some services to k8s, and have had our
> >>> eye on Airflow.
> >>>
> >>>> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <
> sumeet.manit@gmail.com
> >>>
> >>> wrote:
> >>>>
> >>>> I would join as well for sure.
> >>>>
> >>>> Thanks,
> >>>> Sumit Maheshwari
> >>>> cell. 9632202950
> >>>>
> >>>>
> >>>> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
> >> criccomini@apache.org
> >>>>
> >>>> wrote:
> >>>>
> >>>>> I would definitely be up to joining. We're interested in the K8s work
> >>>>> that's going on. That time works for me.
> >>>>>
> >>>>> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> >>>>> daniel.imberman@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hello everyone,
> >>>>>>
> >>>>>> Recently there's been a fair amount of discussion regarding the
> >>>>> integration
> >>>>>> of airflow with kubernetes. If there is interest I would love to
> host
> >>> an
> >>>>>> e-meeting to discuss this integration. I can go over the
> architecture
> >>> as
> >>>>> it
> >>>>>> stands right now and would love feedback on
> >>>>> improvements/features/design. I
> >>>>>> could also attempt to get one or two members of google's kubernetes
> >>> team
> >>>>> to
> >>>>>> join to discuss best practices.
> >>>>>>
> >>>>>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> >>>>> though
> >>>>>> if there's strong opinions otherwise I'd be glad to propose other
> >>> times.
> >>>>>>
> >>>>>> Cheers!
> >>>>>>
> >>>>>> Daniel
> >>>>>>
> >>>>>
> >>>
> >>
>
>
Re: Airflow + Kubernetes discussion
Posted by Bolke de Bruin <bd...@gmail.com>.
Invite would be nice, I will try to join!
> On 20 Jul 2017, at 20:36, Gerard Toonstra <gt...@gmail.com> wrote:
>
> send me an invite too!
>
> On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin <jl...@apache.org> wrote:
>
>> I'm interested as well.
>>
>> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger <ma...@lumoslabs.com> wrote:
>>
>>> +1 We're in the middle of moving some services to k8s, and have had our
>>> eye on Airflow.
>>>
>>>> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <sumeet.manit@gmail.com
>>>
>>> wrote:
>>>>
>>>> I would join as well for sure.
>>>>
>>>> Thanks,
>>>> Sumit Maheshwari
>>>> cell. 9632202950
>>>>
>>>>
>>>> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
>> criccomini@apache.org
>>>>
>>>> wrote:
>>>>
>>>>> I would definitely be up to joining. We're interested in the K8s work
>>>>> that's going on. That time works for me.
>>>>>
>>>>> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
>>>>> daniel.imberman@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> Recently there's been a fair amount of discussion regarding the
>>>>> integration
>>>>>> of airflow with kubernetes. If there is interest I would love to host
>>> an
>>>>>> e-meeting to discuss this integration. I can go over the architecture
>>> as
>>>>> it
>>>>>> stands right now and would love feedback on
>>>>> improvements/features/design. I
>>>>>> could also attempt to get one or two members of google's kubernetes
>>> team
>>>>> to
>>>>>> join to discuss best practices.
>>>>>>
>>>>>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
>>>>> though
>>>>>> if there's strong opinions otherwise I'd be glad to propose other
>>> times.
>>>>>>
>>>>>> Cheers!
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>
>>>
>>
Re: Airflow + Kubernetes discussion
Posted by Gerard Toonstra <gt...@gmail.com>.
send me an invite too!
On Thu, Jul 20, 2017 at 8:17 PM, Jeremiah Lowin <jl...@apache.org> wrote:
> I'm interested as well.
>
> On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger <ma...@lumoslabs.com> wrote:
>
> > +1 We're in the middle of moving some services to k8s, and have had our
> > eye on Airflow.
> >
> > > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <sumeet.manit@gmail.com
> >
> > wrote:
> > >
> > > I would join as well for sure.
> > >
> > > Thanks,
> > > Sumit Maheshwari
> > > cell. 9632202950
> > >
> > >
> > > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <
> criccomini@apache.org
> > >
> > > wrote:
> > >
> > >> I would definitely be up to joining. We're interested in the K8s work
> > >> that's going on. That time works for me.
> > >>
> > >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> > >> daniel.imberman@gmail.com>
> > >> wrote:
> > >>
> > >>> Hello everyone,
> > >>>
> > >>> Recently there's been a fair amount of discussion regarding the
> > >> integration
> > >>> of airflow with kubernetes. If there is interest I would love to host
> > an
> > >>> e-meeting to discuss this integration. I can go over the architecture
> > as
> > >> it
> > >>> stands right now and would love feedback on
> > >> improvements/features/design. I
> > >>> could also attempt to get one or two members of google's kubernetes
> > team
> > >> to
> > >>> join to discuss best practices.
> > >>>
> > >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> > >> though
> > >>> if there's strong opinions otherwise I'd be glad to propose other
> > times.
> > >>>
> > >>> Cheers!
> > >>>
> > >>> Daniel
> > >>>
> > >>
> >
>
Re: Airflow + Kubernetes discussion
Posted by Jeremiah Lowin <jl...@apache.org>.
I'm interested as well.
On Thu, Jul 20, 2017 at 1:51 PM Marc Bollinger <ma...@lumoslabs.com> wrote:
> +1 We're in the middle of moving some services to k8s, and have had our
> eye on Airflow.
>
> > On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <su...@gmail.com>
> wrote:
> >
> > I would join as well for sure.
> >
> > Thanks,
> > Sumit Maheshwari
> > cell. 9632202950
> >
> >
> > On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> >> I would definitely be up to joining. We're interested in the K8s work
> >> that's going on. That time works for me.
> >>
> >> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> >> daniel.imberman@gmail.com>
> >> wrote:
> >>
> >>> Hello everyone,
> >>>
> >>> Recently there's been a fair amount of discussion regarding the
> >> integration
> >>> of airflow with kubernetes. If there is interest I would love to host
> an
> >>> e-meeting to discuss this integration. I can go over the architecture
> as
> >> it
> >>> stands right now and would love feedback on
> >> improvements/features/design. I
> >>> could also attempt to get one or two members of google's kubernetes
> team
> >> to
> >>> join to discuss best practices.
> >>>
> >>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> >> though
> >>> if there's strong opinions otherwise I'd be glad to propose other
> times.
> >>>
> >>> Cheers!
> >>>
> >>> Daniel
> >>>
> >>
>
Re: Airflow + Kubernetes discussion
Posted by Marc Bollinger <ma...@lumoslabs.com>.
+1 We're in the middle of moving some services to k8s, and have had our eye on Airflow.
> On Jul 20, 2017, at 10:37 AM, Sumit Maheshwari <su...@gmail.com> wrote:
>
> I would join as well for sure.
>
> Thanks,
> Sumit Maheshwari
> cell. 9632202950
>
>
> On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <cr...@apache.org>
> wrote:
>
>> I would definitely be up to joining. We're interested in the K8s work
>> that's going on. That time works for me.
>>
>> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
>> daniel.imberman@gmail.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> Recently there's been a fair amount of discussion regarding the
>> integration
>>> of airflow with kubernetes. If there is interest I would love to host an
>>> e-meeting to discuss this integration. I can go over the architecture as
>> it
>>> stands right now and would love feedback on
>> improvements/features/design. I
>>> could also attempt to get one or two members of google's kubernetes team
>> to
>>> join to discuss best practices.
>>>
>>> I'm currently thinking that next Thursday at 11AM PST over zoom.us,
>> though
>>> if there's strong opinions otherwise I'd be glad to propose other times.
>>>
>>> Cheers!
>>>
>>> Daniel
>>>
>>
Re: Airflow + Kubernetes discussion
Posted by Sumit Maheshwari <su...@gmail.com>.
I would join as well for sure.
Thanks,
Sumit Maheshwari
cell. 9632202950
On Thu, Jul 20, 2017 at 11:00 PM, Chris Riccomini <cr...@apache.org>
wrote:
> I would definitely be up to joining. We're interested in the K8s work
> that's going on. That time works for me.
>
> On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > Recently there's been a fair amount of discussion regarding the
> integration
> > of airflow with kubernetes. If there is interest I would love to host an
> > e-meeting to discuss this integration. I can go over the architecture as
> it
> > stands right now and would love feedback on
> improvements/features/design. I
> > could also attempt to get one or two members of google's kubernetes team
> to
> > join to discuss best practices.
> >
> > I'm currently thinking that next Thursday at 11AM PST over zoom.us,
> though
> > if there's strong opinions otherwise I'd be glad to propose other times.
> >
> > Cheers!
> >
> > Daniel
> >
>
Re: Airflow + Kubernetes discussion
Posted by Chris Riccomini <cr...@apache.org>.
I would definitely be up to joining. We're interested in the K8s work
that's going on. That time works for me.
On Thu, Jul 20, 2017 at 9:54 AM, Daniel Imberman <da...@gmail.com>
wrote:
> Hello everyone,
>
> Recently there's been a fair amount of discussion regarding the integration
> of airflow with kubernetes. If there is interest I would love to host an
> e-meeting to discuss this integration. I can go over the architecture as it
> stands right now and would love feedback on improvements/features/design. I
> could also attempt to get one or two members of google's kubernetes team to
> join to discuss best practices.
>
> I'm currently thinking that next Thursday at 11AM PST over zoom.us, though
> if there's strong opinions otherwise I'd be glad to propose other times.
>
> Cheers!
>
> Daniel
>