You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Szymon Przedwojski <sz...@polidea.com> on 2019/05/10 15:22:47 UTC
Re: Bay Area Airflow meetup
Hi Feng,
I'd like to submit my talk proposal for the Bay Area meetup.
*Working title:* Migrating Apache Oozie Workflows to Apache Airflow - the
developer’s perspective
I want to talk about the Oozie-to-Airflow [1] project created by Google
that we at Polidea help develop.
It's a conversion tool written in Python that generates Airflow Python DAGs
from Oozie workflow XMLs, thus enabling easy migration of Oozie workflows
to Airflow.
I would like to present the architecture of the solution and of the whole
end-to-end system tests suite, talk about the challenges we faced when
translating various Oozie features to Airflow and discuss the future of the
project.
Moreover, I will show how the Airflow community can get involved in the
open-source Oozie-to-Airflow project and why it would be mutually
beneficial.
*About me:*
I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
professionally for 5 years now, mainly using Python, Java and JS.
I have public speaking experience - I gave a couple of talks at Python and
JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks at
companies I’ve worked for.
I’ve been involved with Airflow since August last year and I’ve contributed
several operators for different GCP services.
[1] https://github.com/GoogleCloudPlatform/cloud-composer
*Szymon Przedwojski*
Polidea | Software Engineer
M: +48 500 330 790
On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid> wrote:
> We've finalized the meetup logistics (details here
> <
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> >).
> Please send in your talk proposals!
> Thank you.
>
> Feng
>
> On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
>
> > Hi all,
> >
> > It has been a while since we had the last Bay Area meetup and we (Google)
> > are going to host another meetup around June this year. Will share with
> > everyone the logistic details, meanwhile, please feel free to send in
> your
> > talk proposals!
> > (I see a lot of very interesting AIP ideas floating around, these are
> > great candidates.)
> >
> > Looking forward to seeing you all at the event ;)
> >
> > Feng
> >
>
Re: Bay Area Airflow meetup
Posted by Feng Lu <fe...@google.com.INVALID>.
Fantastic Viraj, thanks for sending in your talk. I'll start to compile the
talks I've received.
*From: *Viraj Parekh <vi...@astronomer.io>
*Date: *Fri, May 10, 2019 at 10:45 AM
*To: * <de...@airflow.apache.org>
*Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
Hi Feng,
>
> I'd also like to submit a talk proposal for the Bay Area meetup, works out
> nicely that I'll be in town that week.
>
> Title: Running Apache Airflow Reliably on Kubernetes (and other open source
> software)
>
> Description: As companies scale up their Airflow usage, they need more
> control and observability. By deploying an Airflow stack via Helm on
> Kubernetes, fresh environments can be easily spun up or down, and can scale
> to near 0 when no jobs are running. This talk will go through the technical
> challenges to support thousands of Airflow deployments, how to monitor
> them, reliably push DAGs, and build supporting cloud-native infrastructuret
> using open-source.
>
> About me: I'm a Senior Data Engineer at Astronomer where I've been working
> with Airflow for 2.5 years, and also a Venture for America Fellow. I spend
> most of my time with customer implementations (in all clouds), and
> providing customized Airflow training
>
> On Fri, May 10, 2019 at 1:15 PM Gabriel Silk <gs...@dropbox.com.invalid>
> wrote:
>
> > Hey Feng,
> >
> > I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to talk
> > about the challenges and opportunities in migrating large-scale ETL to
> > Airflow from a system that was built in-house.
> >
> >
> > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > szymon.przedwojski@polidea.com> wrote:
> >
> > > Hi Feng,
> > >
> > > I'd like to submit my talk proposal for the Bay Area meetup.
> > >
> > > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow -
> the
> > > developer’s perspective
> > >
> > > I want to talk about the Oozie-to-Airflow [1] project created by Google
> > > that we at Polidea help develop.
> > > It's a conversion tool written in Python that generates Airflow Python
> > DAGs
> > > from Oozie workflow XMLs, thus enabling easy migration of Oozie
> workflows
> > > to Airflow.
> > > I would like to present the architecture of the solution and of the
> whole
> > > end-to-end system tests suite, talk about the challenges we faced when
> > > translating various Oozie features to Airflow and discuss the future of
> > the
> > > project.
> > > Moreover, I will show how the Airflow community can get involved in the
> > > open-source Oozie-to-Airflow project and why it would be mutually
> > > beneficial.
> > >
> > > *About me:*
> > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> > > professionally for 5 years now, mainly using Python, Java and JS.
> > > I have public speaking experience - I gave a couple of talks at Python
> > and
> > > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks
> at
> > > companies I’ve worked for.
> > > I’ve been involved with Airflow since August last year and I’ve
> > contributed
> > > several operators for different GCP services.
> > >
> > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > >
> > > *Szymon Przedwojski*
> > > Polidea | Software Engineer
> > >
> > > M: +48 500 330 790
> > >
> > >
> > > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid>
> wrote:
> > >
> > > > We've finalized the meetup logistics (details here
> > > > <
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > >).
> > > > Please send in your talk proposals!
> > > > Thank you.
> > > >
> > > > Feng
> > > >
> > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > It has been a while since we had the last Bay Area meetup and we
> > > (Google)
> > > > > are going to host another meetup around June this year. Will share
> > with
> > > > > everyone the logistic details, meanwhile, please feel free to send
> in
> > > > your
> > > > > talk proposals!
> > > > > (I see a lot of very interesting AIP ideas floating around, these
> are
> > > > > great candidates.)
> > > > >
> > > > > Looking forward to seeing you all at the event ;)
> > > > >
> > > > > Feng
> > > > >
> > > >
> > >
> >
>
Re: Bay Area Airflow meetup
Posted by Viraj Parekh <vi...@astronomer.io>.
Hi Feng,
I'd also like to submit a talk proposal for the Bay Area meetup, works out
nicely that I'll be in town that week.
Title: Running Apache Airflow Reliably on Kubernetes (and other open source
software)
Description: As companies scale up their Airflow usage, they need more
control and observability. By deploying an Airflow stack via Helm on
Kubernetes, fresh environments can be easily spun up or down, and can scale
to near 0 when no jobs are running. This talk will go through the technical
challenges to support thousands of Airflow deployments, how to monitor
them, reliably push DAGs, and build supporting cloud-native infrastructuret
using open-source.
About me: I'm a Senior Data Engineer at Astronomer where I've been working
with Airflow for 2.5 years, and also a Venture for America Fellow. I spend
most of my time with customer implementations (in all clouds), and
providing customized Airflow training
On Fri, May 10, 2019 at 1:15 PM Gabriel Silk <gs...@dropbox.com.invalid>
wrote:
> Hey Feng,
>
> I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to talk
> about the challenges and opportunities in migrating large-scale ETL to
> Airflow from a system that was built in-house.
>
>
> On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> szymon.przedwojski@polidea.com> wrote:
>
> > Hi Feng,
> >
> > I'd like to submit my talk proposal for the Bay Area meetup.
> >
> > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow - the
> > developer’s perspective
> >
> > I want to talk about the Oozie-to-Airflow [1] project created by Google
> > that we at Polidea help develop.
> > It's a conversion tool written in Python that generates Airflow Python
> DAGs
> > from Oozie workflow XMLs, thus enabling easy migration of Oozie workflows
> > to Airflow.
> > I would like to present the architecture of the solution and of the whole
> > end-to-end system tests suite, talk about the challenges we faced when
> > translating various Oozie features to Airflow and discuss the future of
> the
> > project.
> > Moreover, I will show how the Airflow community can get involved in the
> > open-source Oozie-to-Airflow project and why it would be mutually
> > beneficial.
> >
> > *About me:*
> > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> > professionally for 5 years now, mainly using Python, Java and JS.
> > I have public speaking experience - I gave a couple of talks at Python
> and
> > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks at
> > companies I’ve worked for.
> > I’ve been involved with Airflow since August last year and I’ve
> contributed
> > several operators for different GCP services.
> >
> > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> >
> > *Szymon Przedwojski*
> > Polidea | Software Engineer
> >
> > M: +48 500 330 790
> >
> >
> > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid> wrote:
> >
> > > We've finalized the meetup logistics (details here
> > > <
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > >).
> > > Please send in your talk proposals!
> > > Thank you.
> > >
> > > Feng
> > >
> > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > It has been a while since we had the last Bay Area meetup and we
> > (Google)
> > > > are going to host another meetup around June this year. Will share
> with
> > > > everyone the logistic details, meanwhile, please feel free to send in
> > > your
> > > > talk proposals!
> > > > (I see a lot of very interesting AIP ideas floating around, these are
> > > > great candidates.)
> > > >
> > > > Looking forward to seeing you all at the event ;)
> > > >
> > > > Feng
> > > >
> > >
> >
>
Re: Bay Area Airflow meetup
Posted by Daniel Standish <dp...@gmail.com>.
Per Feng, the presentations will be recorded and links published to dev
list / meetup page, but will not be live streamed.
On Tue, Jun 4, 2019 at 4:07 PM Teresa Martyny <
teresa.martyny@omadahealth.com> wrote:
> Recording/streaming the meetup talks every time would be amazing for folks
> like me who have kids and can't come to the evening events.
>
> *Teresa Martyny*
> pronouns: she, her, hers
> Software Engineer | Data Team Lead | Omada Health
> <https://www.omadahealth.com/>
> 500 Sansome St #200, SF, CA 94111
>
> *What is Omada?* <https://vimeo.com/203386025>
>
>
> On Tue, Jun 4, 2019 at 3:59 PM Austin Weaver <au...@flyrlabs.com> wrote:
>
> > Thanks for the reminder! Unfortunately the same time as the warriors are
> > playing the nba finals in oakland ;(
> >
> > Is there any chance of this being recorded?
> >
> > Thanks!
> > Austin
> >
> > On Tue, Jun 4, 2019 at 3:40 PM Feng Lu <fe...@google.com.invalid>
> wrote:
> >
> > > A gentle reminder, the meetup will be tomorrow and we just updated the
> > > event
> > > location
> > > <
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > >
> > > to accommodate more people.
> > > See you there.
> > >
> > > On Thu, May 23, 2019 at 11:08 AM Aizhamal Nurmamat kyzy
> > > <ai...@google.com.invalid> wrote:
> > >
> > > > Great list of speakers and topics! Will there be a live-streaming?
> > > >
> > > > Looking forward to the talks :)
> > > >
> > > > On Wed, May 22, 2019 at 11:48 PM Feng Lu <fe...@google.com.invalid>
> > > > wrote:
> > > >
> > > > > We have now finalized the meetup schedule and logistics, lots of
> > > > > interesting talks, please check out the details
> > > > > <
> > > > >
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > > >
> > > > > here.
> > > > > Also a big thank you to all the presenters, see you soon!
> > > > >
> > > > > Feng
> > > > >
> > > > > On Fri, May 10, 2019 at 3:54 PM Feng Lu <fe...@google.com> wrote:
> > > > >
> > > > > > Great, thank you Gabriel.
> > > > > >
> > > > > > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > > > > *Date: *Fri, May 10, 2019 at 1:44 PM
> > > > > > *To: * <de...@airflow.apache.org>
> > > > > > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > > > > >
> > > > > > Sure, how about:
> > > > > >>
> > > > > >> "Migrating Dropbox to Airflow, and the challenges of ETL at
> scale"
> > > > > >>
> > > > > >> Abstract: Dropbox has over 20k batch jobs that run on an hourly
> > > and/or
> > > > > >> daily cadence, connected together in a large web of
> dependencies.
> > > > > >> Migrating
> > > > > >> this ETL to Airflow has proven to be an interesting challenge.
> In
> > > this
> > > > > >> talk
> > > > > >> we'll discover how we went about it, what challenges we are
> > facing,
> > > > how
> > > > > >> Airflow is helping us, and opportunities we see ahead"
> > > > > >>
> > > > > >> On Fri, May 10, 2019 at 1:05 PM Feng Lu
> <fenglu@google.com.invalid
> > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Gabriel,
> > > > > >> >
> > > > > >> > Awesome, could you share with us the talk title and abstract?
> > > > > >> > Thanks.
> > > > > >> >
> > > > > >> > Feng
> > > > > >> >
> > > > > >> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > > > >> > *Date: *Fri, May 10, 2019 at 10:15 AM
> > > > > >> > *To: * <de...@airflow.apache.org>
> > > > > >> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > > > > >> >
> > > > > >> > Hey Feng,
> > > > > >> > >
> > > > > >> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd
> > > like
> > > > to
> > > > > >> talk
> > > > > >> > > about the challenges and opportunities in migrating
> > large-scale
> > > > ETL
> > > > > to
> > > > > >> > > Airflow from a system that was built in-house.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > > > > >> > > szymon.przedwojski@polidea.com> wrote:
> > > > > >> > >
> > > > > >> > > > Hi Feng,
> > > > > >> > > >
> > > > > >> > > > I'd like to submit my talk proposal for the Bay Area
> meetup.
> > > > > >> > > >
> > > > > >> > > > *Working title:* Migrating Apache Oozie Workflows to
> Apache
> > > > > Airflow
> > > > > >> -
> > > > > >> > the
> > > > > >> > > > developer’s perspective
> > > > > >> > > >
> > > > > >> > > > I want to talk about the Oozie-to-Airflow [1] project
> > created
> > > by
> > > > > >> Google
> > > > > >> > > > that we at Polidea help develop.
> > > > > >> > > > It's a conversion tool written in Python that generates
> > > Airflow
> > > > > >> Python
> > > > > >> > > DAGs
> > > > > >> > > > from Oozie workflow XMLs, thus enabling easy migration of
> > > Oozie
> > > > > >> > workflows
> > > > > >> > > > to Airflow.
> > > > > >> > > > I would like to present the architecture of the solution
> and
> > > of
> > > > > the
> > > > > >> > whole
> > > > > >> > > > end-to-end system tests suite, talk about the challenges
> we
> > > > faced
> > > > > >> when
> > > > > >> > > > translating various Oozie features to Airflow and discuss
> > the
> > > > > >> future of
> > > > > >> > > the
> > > > > >> > > > project.
> > > > > >> > > > Moreover, I will show how the Airflow community can get
> > > involved
> > > > > in
> > > > > >> the
> > > > > >> > > > open-source Oozie-to-Airflow project and why it would be
> > > > mutually
> > > > > >> > > > beneficial.
> > > > > >> > > >
> > > > > >> > > > *About me:*
> > > > > >> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been
> > > > programming
> > > > > >> > > > professionally for 5 years now, mainly using Python, Java
> > and
> > > > JS.
> > > > > >> > > > I have public speaking experience - I gave a couple of
> talks
> > > at
> > > > > >> Python
> > > > > >> > > and
> > > > > >> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple
> > > internal
> > > > > >> talks
> > > > > >> > at
> > > > > >> > > > companies I’ve worked for.
> > > > > >> > > > I’ve been involved with Airflow since August last year and
> > > I’ve
> > > > > >> > > contributed
> > > > > >> > > > several operators for different GCP services.
> > > > > >> > > >
> > > > > >> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > > > > >> > > >
> > > > > >> > > > *Szymon Przedwojski*
> > > > > >> > > > Polidea | Software Engineer
> > > > > >> > > >
> > > > > >> > > > M: +48 500 330 790
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu
> > > <fenglu@google.com.invalid
> > > > >
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > > We've finalized the meetup logistics (details here
> > > > > >> > > > > <
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > > >> > > > > >).
> > > > > >> > > > > Please send in your talk proposals!
> > > > > >> > > > > Thank you.
> > > > > >> > > > >
> > > > > >> > > > > Feng
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <
> > fenglu@google.com>
> > > > > >> wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi all,
> > > > > >> > > > > >
> > > > > >> > > > > > It has been a while since we had the last Bay Area
> > meetup
> > > > and
> > > > > we
> > > > > >> > > > (Google)
> > > > > >> > > > > > are going to host another meetup around June this
> year.
> > > Will
> > > > > >> share
> > > > > >> > > with
> > > > > >> > > > > > everyone the logistic details, meanwhile, please feel
> > free
> > > > to
> > > > > >> send
> > > > > >> > in
> > > > > >> > > > > your
> > > > > >> > > > > > talk proposals!
> > > > > >> > > > > > (I see a lot of very interesting AIP ideas floating
> > > around,
> > > > > >> these
> > > > > >> > are
> > > > > >> > > > > > great candidates.)
> > > > > >> > > > > >
> > > > > >> > > > > > Looking forward to seeing you all at the event ;)
> > > > > >> > > > > >
> > > > > >> > > > > > Feng
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Austin Weaver
> > Software Engineer
> > FLYR, Inc. www.flyrlabs.com
> >
>
> --
> This email may contain material that is confidential and/or privileged for
> the sole use of the intended recipient. Any review, reliance, or
> distribution by others or forwarding without express permission is
> strictly
> prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies. Also note that email is not an appropriate
> way to send protected health information to Omada Health employees. Please
> use your discretion when responding to this email.
>
Re: Bay Area Airflow meetup
Posted by Teresa Martyny <te...@omadahealth.com>.
Recording/streaming the meetup talks every time would be amazing for folks
like me who have kids and can't come to the evening events.
*Teresa Martyny*
pronouns: she, her, hers
Software Engineer | Data Team Lead | Omada Health
<https://www.omadahealth.com/>
500 Sansome St #200, SF, CA 94111
*What is Omada?* <https://vimeo.com/203386025>
On Tue, Jun 4, 2019 at 3:59 PM Austin Weaver <au...@flyrlabs.com> wrote:
> Thanks for the reminder! Unfortunately the same time as the warriors are
> playing the nba finals in oakland ;(
>
> Is there any chance of this being recorded?
>
> Thanks!
> Austin
>
> On Tue, Jun 4, 2019 at 3:40 PM Feng Lu <fe...@google.com.invalid> wrote:
>
> > A gentle reminder, the meetup will be tomorrow and we just updated the
> > event
> > location
> > <
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > >
> > to accommodate more people.
> > See you there.
> >
> > On Thu, May 23, 2019 at 11:08 AM Aizhamal Nurmamat kyzy
> > <ai...@google.com.invalid> wrote:
> >
> > > Great list of speakers and topics! Will there be a live-streaming?
> > >
> > > Looking forward to the talks :)
> > >
> > > On Wed, May 22, 2019 at 11:48 PM Feng Lu <fe...@google.com.invalid>
> > > wrote:
> > >
> > > > We have now finalized the meetup schedule and logistics, lots of
> > > > interesting talks, please check out the details
> > > > <
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > >
> > > > here.
> > > > Also a big thank you to all the presenters, see you soon!
> > > >
> > > > Feng
> > > >
> > > > On Fri, May 10, 2019 at 3:54 PM Feng Lu <fe...@google.com> wrote:
> > > >
> > > > > Great, thank you Gabriel.
> > > > >
> > > > > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > > > *Date: *Fri, May 10, 2019 at 1:44 PM
> > > > > *To: * <de...@airflow.apache.org>
> > > > > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > > > >
> > > > > Sure, how about:
> > > > >>
> > > > >> "Migrating Dropbox to Airflow, and the challenges of ETL at scale"
> > > > >>
> > > > >> Abstract: Dropbox has over 20k batch jobs that run on an hourly
> > and/or
> > > > >> daily cadence, connected together in a large web of dependencies.
> > > > >> Migrating
> > > > >> this ETL to Airflow has proven to be an interesting challenge. In
> > this
> > > > >> talk
> > > > >> we'll discover how we went about it, what challenges we are
> facing,
> > > how
> > > > >> Airflow is helping us, and opportunities we see ahead"
> > > > >>
> > > > >> On Fri, May 10, 2019 at 1:05 PM Feng Lu <fenglu@google.com.invalid
> >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Gabriel,
> > > > >> >
> > > > >> > Awesome, could you share with us the talk title and abstract?
> > > > >> > Thanks.
> > > > >> >
> > > > >> > Feng
> > > > >> >
> > > > >> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > > >> > *Date: *Fri, May 10, 2019 at 10:15 AM
> > > > >> > *To: * <de...@airflow.apache.org>
> > > > >> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > > > >> >
> > > > >> > Hey Feng,
> > > > >> > >
> > > > >> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd
> > like
> > > to
> > > > >> talk
> > > > >> > > about the challenges and opportunities in migrating
> large-scale
> > > ETL
> > > > to
> > > > >> > > Airflow from a system that was built in-house.
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > > > >> > > szymon.przedwojski@polidea.com> wrote:
> > > > >> > >
> > > > >> > > > Hi Feng,
> > > > >> > > >
> > > > >> > > > I'd like to submit my talk proposal for the Bay Area meetup.
> > > > >> > > >
> > > > >> > > > *Working title:* Migrating Apache Oozie Workflows to Apache
> > > > Airflow
> > > > >> -
> > > > >> > the
> > > > >> > > > developer’s perspective
> > > > >> > > >
> > > > >> > > > I want to talk about the Oozie-to-Airflow [1] project
> created
> > by
> > > > >> Google
> > > > >> > > > that we at Polidea help develop.
> > > > >> > > > It's a conversion tool written in Python that generates
> > Airflow
> > > > >> Python
> > > > >> > > DAGs
> > > > >> > > > from Oozie workflow XMLs, thus enabling easy migration of
> > Oozie
> > > > >> > workflows
> > > > >> > > > to Airflow.
> > > > >> > > > I would like to present the architecture of the solution and
> > of
> > > > the
> > > > >> > whole
> > > > >> > > > end-to-end system tests suite, talk about the challenges we
> > > faced
> > > > >> when
> > > > >> > > > translating various Oozie features to Airflow and discuss
> the
> > > > >> future of
> > > > >> > > the
> > > > >> > > > project.
> > > > >> > > > Moreover, I will show how the Airflow community can get
> > involved
> > > > in
> > > > >> the
> > > > >> > > > open-source Oozie-to-Airflow project and why it would be
> > > mutually
> > > > >> > > > beneficial.
> > > > >> > > >
> > > > >> > > > *About me:*
> > > > >> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been
> > > programming
> > > > >> > > > professionally for 5 years now, mainly using Python, Java
> and
> > > JS.
> > > > >> > > > I have public speaking experience - I gave a couple of talks
> > at
> > > > >> Python
> > > > >> > > and
> > > > >> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple
> > internal
> > > > >> talks
> > > > >> > at
> > > > >> > > > companies I’ve worked for.
> > > > >> > > > I’ve been involved with Airflow since August last year and
> > I’ve
> > > > >> > > contributed
> > > > >> > > > several operators for different GCP services.
> > > > >> > > >
> > > > >> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > > > >> > > >
> > > > >> > > > *Szymon Przedwojski*
> > > > >> > > > Polidea | Software Engineer
> > > > >> > > >
> > > > >> > > > M: +48 500 330 790
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu
> > <fenglu@google.com.invalid
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > We've finalized the meetup logistics (details here
> > > > >> > > > > <
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > >> > > > > >).
> > > > >> > > > > Please send in your talk proposals!
> > > > >> > > > > Thank you.
> > > > >> > > > >
> > > > >> > > > > Feng
> > > > >> > > > >
> > > > >> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <
> fenglu@google.com>
> > > > >> wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi all,
> > > > >> > > > > >
> > > > >> > > > > > It has been a while since we had the last Bay Area
> meetup
> > > and
> > > > we
> > > > >> > > > (Google)
> > > > >> > > > > > are going to host another meetup around June this year.
> > Will
> > > > >> share
> > > > >> > > with
> > > > >> > > > > > everyone the logistic details, meanwhile, please feel
> free
> > > to
> > > > >> send
> > > > >> > in
> > > > >> > > > > your
> > > > >> > > > > > talk proposals!
> > > > >> > > > > > (I see a lot of very interesting AIP ideas floating
> > around,
> > > > >> these
> > > > >> > are
> > > > >> > > > > > great candidates.)
> > > > >> > > > > >
> > > > >> > > > > > Looking forward to seeing you all at the event ;)
> > > > >> > > > > >
> > > > >> > > > > > Feng
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
>
> --
> Austin Weaver
> Software Engineer
> FLYR, Inc. www.flyrlabs.com
>
--
This email may contain material that is confidential and/or privileged for
the sole use of the intended recipient. Any review, reliance, or
distribution by others or forwarding without express permission is strictly
prohibited. If you are not the intended recipient, please contact the
sender and delete all copies. Also note that email is not an appropriate
way to send protected health information to Omada Health employees. Please
use your discretion when responding to this email.
Re: Bay Area Airflow meetup
Posted by Austin Weaver <au...@flyrlabs.com>.
Thanks for the reminder! Unfortunately the same time as the warriors are
playing the nba finals in oakland ;(
Is there any chance of this being recorded?
Thanks!
Austin
On Tue, Jun 4, 2019 at 3:40 PM Feng Lu <fe...@google.com.invalid> wrote:
> A gentle reminder, the meetup will be tomorrow and we just updated the
> event
> location
> <
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> >
> to accommodate more people.
> See you there.
>
> On Thu, May 23, 2019 at 11:08 AM Aizhamal Nurmamat kyzy
> <ai...@google.com.invalid> wrote:
>
> > Great list of speakers and topics! Will there be a live-streaming?
> >
> > Looking forward to the talks :)
> >
> > On Wed, May 22, 2019 at 11:48 PM Feng Lu <fe...@google.com.invalid>
> > wrote:
> >
> > > We have now finalized the meetup schedule and logistics, lots of
> > > interesting talks, please check out the details
> > > <
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > >
> > > here.
> > > Also a big thank you to all the presenters, see you soon!
> > >
> > > Feng
> > >
> > > On Fri, May 10, 2019 at 3:54 PM Feng Lu <fe...@google.com> wrote:
> > >
> > > > Great, thank you Gabriel.
> > > >
> > > > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > > *Date: *Fri, May 10, 2019 at 1:44 PM
> > > > *To: * <de...@airflow.apache.org>
> > > > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > > >
> > > > Sure, how about:
> > > >>
> > > >> "Migrating Dropbox to Airflow, and the challenges of ETL at scale"
> > > >>
> > > >> Abstract: Dropbox has over 20k batch jobs that run on an hourly
> and/or
> > > >> daily cadence, connected together in a large web of dependencies.
> > > >> Migrating
> > > >> this ETL to Airflow has proven to be an interesting challenge. In
> this
> > > >> talk
> > > >> we'll discover how we went about it, what challenges we are facing,
> > how
> > > >> Airflow is helping us, and opportunities we see ahead"
> > > >>
> > > >> On Fri, May 10, 2019 at 1:05 PM Feng Lu <fe...@google.com.invalid>
> > > >> wrote:
> > > >>
> > > >> > Hi Gabriel,
> > > >> >
> > > >> > Awesome, could you share with us the talk title and abstract?
> > > >> > Thanks.
> > > >> >
> > > >> > Feng
> > > >> >
> > > >> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > >> > *Date: *Fri, May 10, 2019 at 10:15 AM
> > > >> > *To: * <de...@airflow.apache.org>
> > > >> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > > >> >
> > > >> > Hey Feng,
> > > >> > >
> > > >> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd
> like
> > to
> > > >> talk
> > > >> > > about the challenges and opportunities in migrating large-scale
> > ETL
> > > to
> > > >> > > Airflow from a system that was built in-house.
> > > >> > >
> > > >> > >
> > > >> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > > >> > > szymon.przedwojski@polidea.com> wrote:
> > > >> > >
> > > >> > > > Hi Feng,
> > > >> > > >
> > > >> > > > I'd like to submit my talk proposal for the Bay Area meetup.
> > > >> > > >
> > > >> > > > *Working title:* Migrating Apache Oozie Workflows to Apache
> > > Airflow
> > > >> -
> > > >> > the
> > > >> > > > developer’s perspective
> > > >> > > >
> > > >> > > > I want to talk about the Oozie-to-Airflow [1] project created
> by
> > > >> Google
> > > >> > > > that we at Polidea help develop.
> > > >> > > > It's a conversion tool written in Python that generates
> Airflow
> > > >> Python
> > > >> > > DAGs
> > > >> > > > from Oozie workflow XMLs, thus enabling easy migration of
> Oozie
> > > >> > workflows
> > > >> > > > to Airflow.
> > > >> > > > I would like to present the architecture of the solution and
> of
> > > the
> > > >> > whole
> > > >> > > > end-to-end system tests suite, talk about the challenges we
> > faced
> > > >> when
> > > >> > > > translating various Oozie features to Airflow and discuss the
> > > >> future of
> > > >> > > the
> > > >> > > > project.
> > > >> > > > Moreover, I will show how the Airflow community can get
> involved
> > > in
> > > >> the
> > > >> > > > open-source Oozie-to-Airflow project and why it would be
> > mutually
> > > >> > > > beneficial.
> > > >> > > >
> > > >> > > > *About me:*
> > > >> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been
> > programming
> > > >> > > > professionally for 5 years now, mainly using Python, Java and
> > JS.
> > > >> > > > I have public speaking experience - I gave a couple of talks
> at
> > > >> Python
> > > >> > > and
> > > >> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple
> internal
> > > >> talks
> > > >> > at
> > > >> > > > companies I’ve worked for.
> > > >> > > > I’ve been involved with Airflow since August last year and
> I’ve
> > > >> > > contributed
> > > >> > > > several operators for different GCP services.
> > > >> > > >
> > > >> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > > >> > > >
> > > >> > > > *Szymon Przedwojski*
> > > >> > > > Polidea | Software Engineer
> > > >> > > >
> > > >> > > > M: +48 500 330 790
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu
> <fenglu@google.com.invalid
> > >
> > > >> > wrote:
> > > >> > > >
> > > >> > > > > We've finalized the meetup logistics (details here
> > > >> > > > > <
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > >> > > > > >).
> > > >> > > > > Please send in your talk proposals!
> > > >> > > > > Thank you.
> > > >> > > > >
> > > >> > > > > Feng
> > > >> > > > >
> > > >> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > Hi all,
> > > >> > > > > >
> > > >> > > > > > It has been a while since we had the last Bay Area meetup
> > and
> > > we
> > > >> > > > (Google)
> > > >> > > > > > are going to host another meetup around June this year.
> Will
> > > >> share
> > > >> > > with
> > > >> > > > > > everyone the logistic details, meanwhile, please feel free
> > to
> > > >> send
> > > >> > in
> > > >> > > > > your
> > > >> > > > > > talk proposals!
> > > >> > > > > > (I see a lot of very interesting AIP ideas floating
> around,
> > > >> these
> > > >> > are
> > > >> > > > > > great candidates.)
> > > >> > > > > >
> > > >> > > > > > Looking forward to seeing you all at the event ;)
> > > >> > > > > >
> > > >> > > > > > Feng
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
--
Austin Weaver
Software Engineer
FLYR, Inc. www.flyrlabs.com
Re: Bay Area Airflow meetup
Posted by Feng Lu <fe...@google.com.INVALID>.
A gentle reminder, the meetup will be tomorrow and we just updated the event
location
<https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/>
to accommodate more people.
See you there.
On Thu, May 23, 2019 at 11:08 AM Aizhamal Nurmamat kyzy
<ai...@google.com.invalid> wrote:
> Great list of speakers and topics! Will there be a live-streaming?
>
> Looking forward to the talks :)
>
> On Wed, May 22, 2019 at 11:48 PM Feng Lu <fe...@google.com.invalid>
> wrote:
>
> > We have now finalized the meetup schedule and logistics, lots of
> > interesting talks, please check out the details
> > <
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > >
> > here.
> > Also a big thank you to all the presenters, see you soon!
> >
> > Feng
> >
> > On Fri, May 10, 2019 at 3:54 PM Feng Lu <fe...@google.com> wrote:
> >
> > > Great, thank you Gabriel.
> > >
> > > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > > *Date: *Fri, May 10, 2019 at 1:44 PM
> > > *To: * <de...@airflow.apache.org>
> > > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > >
> > > Sure, how about:
> > >>
> > >> "Migrating Dropbox to Airflow, and the challenges of ETL at scale"
> > >>
> > >> Abstract: Dropbox has over 20k batch jobs that run on an hourly and/or
> > >> daily cadence, connected together in a large web of dependencies.
> > >> Migrating
> > >> this ETL to Airflow has proven to be an interesting challenge. In this
> > >> talk
> > >> we'll discover how we went about it, what challenges we are facing,
> how
> > >> Airflow is helping us, and opportunities we see ahead"
> > >>
> > >> On Fri, May 10, 2019 at 1:05 PM Feng Lu <fe...@google.com.invalid>
> > >> wrote:
> > >>
> > >> > Hi Gabriel,
> > >> >
> > >> > Awesome, could you share with us the talk title and abstract?
> > >> > Thanks.
> > >> >
> > >> > Feng
> > >> >
> > >> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > >> > *Date: *Fri, May 10, 2019 at 10:15 AM
> > >> > *To: * <de...@airflow.apache.org>
> > >> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> > >> >
> > >> > Hey Feng,
> > >> > >
> > >> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd like
> to
> > >> talk
> > >> > > about the challenges and opportunities in migrating large-scale
> ETL
> > to
> > >> > > Airflow from a system that was built in-house.
> > >> > >
> > >> > >
> > >> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > >> > > szymon.przedwojski@polidea.com> wrote:
> > >> > >
> > >> > > > Hi Feng,
> > >> > > >
> > >> > > > I'd like to submit my talk proposal for the Bay Area meetup.
> > >> > > >
> > >> > > > *Working title:* Migrating Apache Oozie Workflows to Apache
> > Airflow
> > >> -
> > >> > the
> > >> > > > developer’s perspective
> > >> > > >
> > >> > > > I want to talk about the Oozie-to-Airflow [1] project created by
> > >> Google
> > >> > > > that we at Polidea help develop.
> > >> > > > It's a conversion tool written in Python that generates Airflow
> > >> Python
> > >> > > DAGs
> > >> > > > from Oozie workflow XMLs, thus enabling easy migration of Oozie
> > >> > workflows
> > >> > > > to Airflow.
> > >> > > > I would like to present the architecture of the solution and of
> > the
> > >> > whole
> > >> > > > end-to-end system tests suite, talk about the challenges we
> faced
> > >> when
> > >> > > > translating various Oozie features to Airflow and discuss the
> > >> future of
> > >> > > the
> > >> > > > project.
> > >> > > > Moreover, I will show how the Airflow community can get involved
> > in
> > >> the
> > >> > > > open-source Oozie-to-Airflow project and why it would be
> mutually
> > >> > > > beneficial.
> > >> > > >
> > >> > > > *About me:*
> > >> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been
> programming
> > >> > > > professionally for 5 years now, mainly using Python, Java and
> JS.
> > >> > > > I have public speaking experience - I gave a couple of talks at
> > >> Python
> > >> > > and
> > >> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal
> > >> talks
> > >> > at
> > >> > > > companies I’ve worked for.
> > >> > > > I’ve been involved with Airflow since August last year and I’ve
> > >> > > contributed
> > >> > > > several operators for different GCP services.
> > >> > > >
> > >> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > >> > > >
> > >> > > > *Szymon Przedwojski*
> > >> > > > Polidea | Software Engineer
> > >> > > >
> > >> > > > M: +48 500 330 790
> > >> > > >
> > >> > > >
> > >> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fenglu@google.com.invalid
> >
> > >> > wrote:
> > >> > > >
> > >> > > > > We've finalized the meetup logistics (details here
> > >> > > > > <
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > >> > > > > >).
> > >> > > > > Please send in your talk proposals!
> > >> > > > > Thank you.
> > >> > > > >
> > >> > > > > Feng
> > >> > > > >
> > >> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > It has been a while since we had the last Bay Area meetup
> and
> > we
> > >> > > > (Google)
> > >> > > > > > are going to host another meetup around June this year. Will
> > >> share
> > >> > > with
> > >> > > > > > everyone the logistic details, meanwhile, please feel free
> to
> > >> send
> > >> > in
> > >> > > > > your
> > >> > > > > > talk proposals!
> > >> > > > > > (I see a lot of very interesting AIP ideas floating around,
> > >> these
> > >> > are
> > >> > > > > > great candidates.)
> > >> > > > > >
> > >> > > > > > Looking forward to seeing you all at the event ;)
> > >> > > > > >
> > >> > > > > > Feng
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Bay Area Airflow meetup
Posted by Aizhamal Nurmamat kyzy <ai...@google.com.INVALID>.
Great list of speakers and topics! Will there be a live-streaming?
Looking forward to the talks :)
On Wed, May 22, 2019 at 11:48 PM Feng Lu <fe...@google.com.invalid> wrote:
> We have now finalized the meetup schedule and logistics, lots of
> interesting talks, please check out the details
> <
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> >
> here.
> Also a big thank you to all the presenters, see you soon!
>
> Feng
>
> On Fri, May 10, 2019 at 3:54 PM Feng Lu <fe...@google.com> wrote:
>
> > Great, thank you Gabriel.
> >
> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > *Date: *Fri, May 10, 2019 at 1:44 PM
> > *To: * <de...@airflow.apache.org>
> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> >
> > Sure, how about:
> >>
> >> "Migrating Dropbox to Airflow, and the challenges of ETL at scale"
> >>
> >> Abstract: Dropbox has over 20k batch jobs that run on an hourly and/or
> >> daily cadence, connected together in a large web of dependencies.
> >> Migrating
> >> this ETL to Airflow has proven to be an interesting challenge. In this
> >> talk
> >> we'll discover how we went about it, what challenges we are facing, how
> >> Airflow is helping us, and opportunities we see ahead"
> >>
> >> On Fri, May 10, 2019 at 1:05 PM Feng Lu <fe...@google.com.invalid>
> >> wrote:
> >>
> >> > Hi Gabriel,
> >> >
> >> > Awesome, could you share with us the talk title and abstract?
> >> > Thanks.
> >> >
> >> > Feng
> >> >
> >> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> >> > *Date: *Fri, May 10, 2019 at 10:15 AM
> >> > *To: * <de...@airflow.apache.org>
> >> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> >> >
> >> > Hey Feng,
> >> > >
> >> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to
> >> talk
> >> > > about the challenges and opportunities in migrating large-scale ETL
> to
> >> > > Airflow from a system that was built in-house.
> >> > >
> >> > >
> >> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> >> > > szymon.przedwojski@polidea.com> wrote:
> >> > >
> >> > > > Hi Feng,
> >> > > >
> >> > > > I'd like to submit my talk proposal for the Bay Area meetup.
> >> > > >
> >> > > > *Working title:* Migrating Apache Oozie Workflows to Apache
> Airflow
> >> -
> >> > the
> >> > > > developer’s perspective
> >> > > >
> >> > > > I want to talk about the Oozie-to-Airflow [1] project created by
> >> Google
> >> > > > that we at Polidea help develop.
> >> > > > It's a conversion tool written in Python that generates Airflow
> >> Python
> >> > > DAGs
> >> > > > from Oozie workflow XMLs, thus enabling easy migration of Oozie
> >> > workflows
> >> > > > to Airflow.
> >> > > > I would like to present the architecture of the solution and of
> the
> >> > whole
> >> > > > end-to-end system tests suite, talk about the challenges we faced
> >> when
> >> > > > translating various Oozie features to Airflow and discuss the
> >> future of
> >> > > the
> >> > > > project.
> >> > > > Moreover, I will show how the Airflow community can get involved
> in
> >> the
> >> > > > open-source Oozie-to-Airflow project and why it would be mutually
> >> > > > beneficial.
> >> > > >
> >> > > > *About me:*
> >> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> >> > > > professionally for 5 years now, mainly using Python, Java and JS.
> >> > > > I have public speaking experience - I gave a couple of talks at
> >> Python
> >> > > and
> >> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal
> >> talks
> >> > at
> >> > > > companies I’ve worked for.
> >> > > > I’ve been involved with Airflow since August last year and I’ve
> >> > > contributed
> >> > > > several operators for different GCP services.
> >> > > >
> >> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> >> > > >
> >> > > > *Szymon Przedwojski*
> >> > > > Polidea | Software Engineer
> >> > > >
> >> > > > M: +48 500 330 790
> >> > > >
> >> > > >
> >> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid>
> >> > wrote:
> >> > > >
> >> > > > > We've finalized the meetup logistics (details here
> >> > > > > <
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> >> > > > > >).
> >> > > > > Please send in your talk proposals!
> >> > > > > Thank you.
> >> > > > >
> >> > > > > Feng
> >> > > > >
> >> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com>
> >> wrote:
> >> > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > It has been a while since we had the last Bay Area meetup and
> we
> >> > > > (Google)
> >> > > > > > are going to host another meetup around June this year. Will
> >> share
> >> > > with
> >> > > > > > everyone the logistic details, meanwhile, please feel free to
> >> send
> >> > in
> >> > > > > your
> >> > > > > > talk proposals!
> >> > > > > > (I see a lot of very interesting AIP ideas floating around,
> >> these
> >> > are
> >> > > > > > great candidates.)
> >> > > > > >
> >> > > > > > Looking forward to seeing you all at the event ;)
> >> > > > > >
> >> > > > > > Feng
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Bay Area Airflow meetup
Posted by Feng Lu <fe...@google.com.INVALID>.
We have now finalized the meetup schedule and logistics, lots of
interesting talks, please check out the details
<https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/>
here.
Also a big thank you to all the presenters, see you soon!
Feng
On Fri, May 10, 2019 at 3:54 PM Feng Lu <fe...@google.com> wrote:
> Great, thank you Gabriel.
>
> *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> *Date: *Fri, May 10, 2019 at 1:44 PM
> *To: * <de...@airflow.apache.org>
> *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
>
> Sure, how about:
>>
>> "Migrating Dropbox to Airflow, and the challenges of ETL at scale"
>>
>> Abstract: Dropbox has over 20k batch jobs that run on an hourly and/or
>> daily cadence, connected together in a large web of dependencies.
>> Migrating
>> this ETL to Airflow has proven to be an interesting challenge. In this
>> talk
>> we'll discover how we went about it, what challenges we are facing, how
>> Airflow is helping us, and opportunities we see ahead"
>>
>> On Fri, May 10, 2019 at 1:05 PM Feng Lu <fe...@google.com.invalid>
>> wrote:
>>
>> > Hi Gabriel,
>> >
>> > Awesome, could you share with us the talk title and abstract?
>> > Thanks.
>> >
>> > Feng
>> >
>> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
>> > *Date: *Fri, May 10, 2019 at 10:15 AM
>> > *To: * <de...@airflow.apache.org>
>> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
>> >
>> > Hey Feng,
>> > >
>> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to
>> talk
>> > > about the challenges and opportunities in migrating large-scale ETL to
>> > > Airflow from a system that was built in-house.
>> > >
>> > >
>> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
>> > > szymon.przedwojski@polidea.com> wrote:
>> > >
>> > > > Hi Feng,
>> > > >
>> > > > I'd like to submit my talk proposal for the Bay Area meetup.
>> > > >
>> > > > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow
>> -
>> > the
>> > > > developer’s perspective
>> > > >
>> > > > I want to talk about the Oozie-to-Airflow [1] project created by
>> Google
>> > > > that we at Polidea help develop.
>> > > > It's a conversion tool written in Python that generates Airflow
>> Python
>> > > DAGs
>> > > > from Oozie workflow XMLs, thus enabling easy migration of Oozie
>> > workflows
>> > > > to Airflow.
>> > > > I would like to present the architecture of the solution and of the
>> > whole
>> > > > end-to-end system tests suite, talk about the challenges we faced
>> when
>> > > > translating various Oozie features to Airflow and discuss the
>> future of
>> > > the
>> > > > project.
>> > > > Moreover, I will show how the Airflow community can get involved in
>> the
>> > > > open-source Oozie-to-Airflow project and why it would be mutually
>> > > > beneficial.
>> > > >
>> > > > *About me:*
>> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
>> > > > professionally for 5 years now, mainly using Python, Java and JS.
>> > > > I have public speaking experience - I gave a couple of talks at
>> Python
>> > > and
>> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal
>> talks
>> > at
>> > > > companies I’ve worked for.
>> > > > I’ve been involved with Airflow since August last year and I’ve
>> > > contributed
>> > > > several operators for different GCP services.
>> > > >
>> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
>> > > >
>> > > > *Szymon Przedwojski*
>> > > > Polidea | Software Engineer
>> > > >
>> > > > M: +48 500 330 790
>> > > >
>> > > >
>> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid>
>> > wrote:
>> > > >
>> > > > > We've finalized the meetup logistics (details here
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
>> > > > > >).
>> > > > > Please send in your talk proposals!
>> > > > > Thank you.
>> > > > >
>> > > > > Feng
>> > > > >
>> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com>
>> wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > It has been a while since we had the last Bay Area meetup and we
>> > > > (Google)
>> > > > > > are going to host another meetup around June this year. Will
>> share
>> > > with
>> > > > > > everyone the logistic details, meanwhile, please feel free to
>> send
>> > in
>> > > > > your
>> > > > > > talk proposals!
>> > > > > > (I see a lot of very interesting AIP ideas floating around,
>> these
>> > are
>> > > > > > great candidates.)
>> > > > > >
>> > > > > > Looking forward to seeing you all at the event ;)
>> > > > > >
>> > > > > > Feng
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Bay Area Airflow meetup
Posted by Feng Lu <fe...@google.com.INVALID>.
Great, thank you Gabriel.
*From: *Gabriel Silk <gs...@dropbox.com.invalid>
*Date: *Fri, May 10, 2019 at 1:44 PM
*To: * <de...@airflow.apache.org>
*Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
Sure, how about:
>
> "Migrating Dropbox to Airflow, and the challenges of ETL at scale"
>
> Abstract: Dropbox has over 20k batch jobs that run on an hourly and/or
> daily cadence, connected together in a large web of dependencies. Migrating
> this ETL to Airflow has proven to be an interesting challenge. In this talk
> we'll discover how we went about it, what challenges we are facing, how
> Airflow is helping us, and opportunities we see ahead"
>
> On Fri, May 10, 2019 at 1:05 PM Feng Lu <fe...@google.com.invalid> wrote:
>
> > Hi Gabriel,
> >
> > Awesome, could you share with us the talk title and abstract?
> > Thanks.
> >
> > Feng
> >
> > *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> > *Date: *Fri, May 10, 2019 at 10:15 AM
> > *To: * <de...@airflow.apache.org>
> > *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
> >
> > Hey Feng,
> > >
> > > I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to
> talk
> > > about the challenges and opportunities in migrating large-scale ETL to
> > > Airflow from a system that was built in-house.
> > >
> > >
> > > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > > szymon.przedwojski@polidea.com> wrote:
> > >
> > > > Hi Feng,
> > > >
> > > > I'd like to submit my talk proposal for the Bay Area meetup.
> > > >
> > > > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow -
> > the
> > > > developer’s perspective
> > > >
> > > > I want to talk about the Oozie-to-Airflow [1] project created by
> Google
> > > > that we at Polidea help develop.
> > > > It's a conversion tool written in Python that generates Airflow
> Python
> > > DAGs
> > > > from Oozie workflow XMLs, thus enabling easy migration of Oozie
> > workflows
> > > > to Airflow.
> > > > I would like to present the architecture of the solution and of the
> > whole
> > > > end-to-end system tests suite, talk about the challenges we faced
> when
> > > > translating various Oozie features to Airflow and discuss the future
> of
> > > the
> > > > project.
> > > > Moreover, I will show how the Airflow community can get involved in
> the
> > > > open-source Oozie-to-Airflow project and why it would be mutually
> > > > beneficial.
> > > >
> > > > *About me:*
> > > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> > > > professionally for 5 years now, mainly using Python, Java and JS.
> > > > I have public speaking experience - I gave a couple of talks at
> Python
> > > and
> > > > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal
> talks
> > at
> > > > companies I’ve worked for.
> > > > I’ve been involved with Airflow since August last year and I’ve
> > > contributed
> > > > several operators for different GCP services.
> > > >
> > > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > > >
> > > > *Szymon Przedwojski*
> > > > Polidea | Software Engineer
> > > >
> > > > M: +48 500 330 790
> > > >
> > > >
> > > > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid>
> > wrote:
> > > >
> > > > > We've finalized the meetup logistics (details here
> > > > > <
> > > > >
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > > >).
> > > > > Please send in your talk proposals!
> > > > > Thank you.
> > > > >
> > > > > Feng
> > > > >
> > > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > It has been a while since we had the last Bay Area meetup and we
> > > > (Google)
> > > > > > are going to host another meetup around June this year. Will
> share
> > > with
> > > > > > everyone the logistic details, meanwhile, please feel free to
> send
> > in
> > > > > your
> > > > > > talk proposals!
> > > > > > (I see a lot of very interesting AIP ideas floating around, these
> > are
> > > > > > great candidates.)
> > > > > >
> > > > > > Looking forward to seeing you all at the event ;)
> > > > > >
> > > > > > Feng
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Bay Area Airflow meetup
Posted by Gabriel Silk <gs...@dropbox.com.INVALID>.
Sure, how about:
"Migrating Dropbox to Airflow, and the challenges of ETL at scale"
Abstract: Dropbox has over 20k batch jobs that run on an hourly and/or
daily cadence, connected together in a large web of dependencies. Migrating
this ETL to Airflow has proven to be an interesting challenge. In this talk
we'll discover how we went about it, what challenges we are facing, how
Airflow is helping us, and opportunities we see ahead"
On Fri, May 10, 2019 at 1:05 PM Feng Lu <fe...@google.com.invalid> wrote:
> Hi Gabriel,
>
> Awesome, could you share with us the talk title and abstract?
> Thanks.
>
> Feng
>
> *From: *Gabriel Silk <gs...@dropbox.com.invalid>
> *Date: *Fri, May 10, 2019 at 10:15 AM
> *To: * <de...@airflow.apache.org>
> *Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
>
> Hey Feng,
> >
> > I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to talk
> > about the challenges and opportunities in migrating large-scale ETL to
> > Airflow from a system that was built in-house.
> >
> >
> > On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> > szymon.przedwojski@polidea.com> wrote:
> >
> > > Hi Feng,
> > >
> > > I'd like to submit my talk proposal for the Bay Area meetup.
> > >
> > > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow -
> the
> > > developer’s perspective
> > >
> > > I want to talk about the Oozie-to-Airflow [1] project created by Google
> > > that we at Polidea help develop.
> > > It's a conversion tool written in Python that generates Airflow Python
> > DAGs
> > > from Oozie workflow XMLs, thus enabling easy migration of Oozie
> workflows
> > > to Airflow.
> > > I would like to present the architecture of the solution and of the
> whole
> > > end-to-end system tests suite, talk about the challenges we faced when
> > > translating various Oozie features to Airflow and discuss the future of
> > the
> > > project.
> > > Moreover, I will show how the Airflow community can get involved in the
> > > open-source Oozie-to-Airflow project and why it would be mutually
> > > beneficial.
> > >
> > > *About me:*
> > > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> > > professionally for 5 years now, mainly using Python, Java and JS.
> > > I have public speaking experience - I gave a couple of talks at Python
> > and
> > > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks
> at
> > > companies I’ve worked for.
> > > I’ve been involved with Airflow since August last year and I’ve
> > contributed
> > > several operators for different GCP services.
> > >
> > > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> > >
> > > *Szymon Przedwojski*
> > > Polidea | Software Engineer
> > >
> > > M: +48 500 330 790
> > >
> > >
> > > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid>
> wrote:
> > >
> > > > We've finalized the meetup logistics (details here
> > > > <
> > > >
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > > >).
> > > > Please send in your talk proposals!
> > > > Thank you.
> > > >
> > > > Feng
> > > >
> > > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > It has been a while since we had the last Bay Area meetup and we
> > > (Google)
> > > > > are going to host another meetup around June this year. Will share
> > with
> > > > > everyone the logistic details, meanwhile, please feel free to send
> in
> > > > your
> > > > > talk proposals!
> > > > > (I see a lot of very interesting AIP ideas floating around, these
> are
> > > > > great candidates.)
> > > > >
> > > > > Looking forward to seeing you all at the event ;)
> > > > >
> > > > > Feng
> > > > >
> > > >
> > >
> >
>
Re: Bay Area Airflow meetup
Posted by Feng Lu <fe...@google.com.INVALID>.
Hi Gabriel,
Awesome, could you share with us the talk title and abstract?
Thanks.
Feng
*From: *Gabriel Silk <gs...@dropbox.com.invalid>
*Date: *Fri, May 10, 2019 at 10:15 AM
*To: * <de...@airflow.apache.org>
*Cc: *Griselda Cuevas, Leah Cole, Trishka Fernandes
Hey Feng,
>
> I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to talk
> about the challenges and opportunities in migrating large-scale ETL to
> Airflow from a system that was built in-house.
>
>
> On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
> szymon.przedwojski@polidea.com> wrote:
>
> > Hi Feng,
> >
> > I'd like to submit my talk proposal for the Bay Area meetup.
> >
> > *Working title:* Migrating Apache Oozie Workflows to Apache Airflow - the
> > developer’s perspective
> >
> > I want to talk about the Oozie-to-Airflow [1] project created by Google
> > that we at Polidea help develop.
> > It's a conversion tool written in Python that generates Airflow Python
> DAGs
> > from Oozie workflow XMLs, thus enabling easy migration of Oozie workflows
> > to Airflow.
> > I would like to present the architecture of the solution and of the whole
> > end-to-end system tests suite, talk about the challenges we faced when
> > translating various Oozie features to Airflow and discuss the future of
> the
> > project.
> > Moreover, I will show how the Airflow community can get involved in the
> > open-source Oozie-to-Airflow project and why it would be mutually
> > beneficial.
> >
> > *About me:*
> > I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> > professionally for 5 years now, mainly using Python, Java and JS.
> > I have public speaking experience - I gave a couple of talks at Python
> and
> > JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks at
> > companies I’ve worked for.
> > I’ve been involved with Airflow since August last year and I’ve
> contributed
> > several operators for different GCP services.
> >
> > [1] https://github.com/GoogleCloudPlatform/cloud-composer
> >
> > *Szymon Przedwojski*
> > Polidea | Software Engineer
> >
> > M: +48 500 330 790
> >
> >
> > On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid> wrote:
> >
> > > We've finalized the meetup logistics (details here
> > > <
> > >
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > > >).
> > > Please send in your talk proposals!
> > > Thank you.
> > >
> > > Feng
> > >
> > > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > It has been a while since we had the last Bay Area meetup and we
> > (Google)
> > > > are going to host another meetup around June this year. Will share
> with
> > > > everyone the logistic details, meanwhile, please feel free to send in
> > > your
> > > > talk proposals!
> > > > (I see a lot of very interesting AIP ideas floating around, these are
> > > > great candidates.)
> > > >
> > > > Looking forward to seeing you all at the event ;)
> > > >
> > > > Feng
> > > >
> > >
> >
>
Re: Bay Area Airflow meetup
Posted by Gabriel Silk <gs...@dropbox.com.INVALID>.
Hey Feng,
I'm the tech lead on Data Infrastructure at Dropbox, and I'd like to talk
about the challenges and opportunities in migrating large-scale ETL to
Airflow from a system that was built in-house.
On Fri, May 10, 2019 at 8:23 AM Szymon Przedwojski <
szymon.przedwojski@polidea.com> wrote:
> Hi Feng,
>
> I'd like to submit my talk proposal for the Bay Area meetup.
>
> *Working title:* Migrating Apache Oozie Workflows to Apache Airflow - the
> developer’s perspective
>
> I want to talk about the Oozie-to-Airflow [1] project created by Google
> that we at Polidea help develop.
> It's a conversion tool written in Python that generates Airflow Python DAGs
> from Oozie workflow XMLs, thus enabling easy migration of Oozie workflows
> to Airflow.
> I would like to present the architecture of the solution and of the whole
> end-to-end system tests suite, talk about the challenges we faced when
> translating various Oozie features to Airflow and discuss the future of the
> project.
> Moreover, I will show how the Airflow community can get involved in the
> open-source Oozie-to-Airflow project and why it would be mutually
> beneficial.
>
> *About me:*
> I’m a Software Engineer @ Polidea in Warsaw. I’ve been programming
> professionally for 5 years now, mainly using Python, Java and JS.
> I have public speaking experience - I gave a couple of talks at Python and
> JS meetups, at a GDG Dev Fest Warsaw event and multiple internal talks at
> companies I’ve worked for.
> I’ve been involved with Airflow since August last year and I’ve contributed
> several operators for different GCP services.
>
> [1] https://github.com/GoogleCloudPlatform/cloud-composer
>
> *Szymon Przedwojski*
> Polidea | Software Engineer
>
> M: +48 500 330 790
>
>
> On Thu, 18 Apr 2019 at 01:08, Feng Lu <fe...@google.com.invalid> wrote:
>
> > We've finalized the meetup logistics (details here
> > <
> >
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/260712102/
> > >).
> > Please send in your talk proposals!
> > Thank you.
> >
> > Feng
> >
> > On Tue, Mar 12, 2019 at 1:15 PM Feng Lu <fe...@google.com> wrote:
> >
> > > Hi all,
> > >
> > > It has been a while since we had the last Bay Area meetup and we
> (Google)
> > > are going to host another meetup around June this year. Will share with
> > > everyone the logistic details, meanwhile, please feel free to send in
> > your
> > > talk proposals!
> > > (I see a lot of very interesting AIP ideas floating around, these are
> > > great candidates.)
> > >
> > > Looking forward to seeing you all at the event ;)
> > >
> > > Feng
> > >
> >
>