You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Joy Gao <jo...@wepay.com> on 2018/01/05 20:10:15 UTC
Q1 Airflow Bay Area Meetup
Hi folks,
I'm Joy from WePay. At the last Airflow meetup in December there was a
demand for hosting a future meetup to cover topics on Airflow integrations
with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
practices on running Airflow in the cloud, managed Airflow, etc.)
If there is enough interest, WePay can host the next event sometimes in
March/April. And if you would like to give a talk or have another topic
covered, feel free to suggest them as well.
Cheers,
Joy
Re: Q1 Airflow Bay Area Meetup
Posted by Joy Gao <jo...@wepay.com>.
Hi Sumit!
The date has been finalized. That said, I will attempt to live-stream or
record the event for folks who cannot attend in person. Appreciate your
understanding.
Thanks,
Joy
On Thu, Feb 15, 2018 at 12:55 AM, Sumit Maheshwari <ms...@apache.org>
wrote:
> We can present our internal tool for doing data quality checks for ETL's
>> in Qubole.
>> Iris is an Airflow Operator that currently has built-in functions for count
>> validation, data-profile validation and duplicate data validation.
>
>
> That would be great Devjyoti.
>
>
> folks,
>
> I would really love to attend this meetup and meet the Airflow community.
> I'll be attending dataplatforms <https://www.dataplatforms.com/>conference
> between 11-13th Apr and would be in the bay area before and after that.
>
> On the Airflow meetup page, I see that it's scheduled on 11th Apr, so if
> possible can we move it to 13th or some other day, it would be great.
>
>
> Thanks,
> Sumit
>
>
> On Wed, Jan 31, 2018 at 1:52 PM, Devjyoti Patra <de...@qubole.com>
> wrote:
>
>> We can present our internal tool for doing data quality checks for ETL's
>> in
>> Qubole.
>> Iris is an Airflow Operator that currently has built-in functions for
>> count validation,
>> data-profile validation and duplicate data validation.
>>
>> On Wed, Jan 31, 2018 at 3:10 AM, George Leslie-Waksman <
>> george@cloverhealth.com.invalid> wrote:
>>
>> > If you're still looking for speakers, I have a handful of things I could
>> > pull together to talk about:
>> >
>> > - the interplay between pools, queues, parallelism, dag concurrency,
>> and
>> > other job scheduling levers
>> > - layering additional instrumentation on top of Airflow operators
>> > - integrity testing DAGs in CI before they hit production / unit
>> testing
>> > tasks
>> > - using Airflow metadata to track systemic execution behavior
>> > - how I would have used Airflow if I'd known then what I know now
>> >
>> >
>> > On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <an...@gmail.com>
>> > wrote:
>> >
>> > > +1 on being able to watch a recording, based in Dublin, Ireland but
>> also
>> > > very interested in the cloud side of airflow.
>> > >
>> > > On Thu, 11 Jan 2018, 17:59 Joy Gao, <jo...@wepay.com> wrote:
>> > >
>> > > > Great to see that there's a lot of interests!
>> > > >
>> > > > I will go ahead and work on organizing this event. Details TBD.
>> > > >
>> > > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <
>> > llorenz@industrydive.com>
>> > > > wrote:
>> > > >
>> > > > > I would love to see videos from this as well, as we're east
>> coast. We
>> > > > host
>> > > > > our airflow installation in GCP using GKE, and would love to
>> compare
>> > > > notes.
>> > > > >
>> > > > > Laura
>> > > > >
>> > > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bdbruin@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > If it coincides with qcon.ai (april 10-11) or is close to its
>> > dates
>> > > > > then I
>> > > > > > can join.
>> > > > > >
>> > > > > > B.
>> > > > > >
>> > > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <
>> > vananth22@gmail.com
>> > > >:
>> > > > > >
>> > > > > > I can give a talk about all the hacks we did to scale Airflow
>> Local
>> > > > > > Executor and improve the data pipeline on-call experience at
>> Slack
>> > if
>> > > > > folks
>> > > > > > are interested.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Ananth.P,
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 8 January 2018 at 15:51, George Leslie-Waksman <
>> > > > > > george@cloverhealth.com.invalid> wrote:
>> > > > > >
>> > > > > > > +1 and would love to hear about other folks running Airflow in
>> > GCP.
>> > > > > > >
>> > > > > > > Would strongly prefer March (but I'm just one person)
>> > > > > > >
>> > > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu
>> <fenglu@google.com.invalid
>> > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > +1 Joy!
>> > > > > > > >
>> > > > > > > > Would prefer April if possible, I can talk about running
>> > Airflow
>> > > in
>> > > > > GCP
>> > > > > > > if
>> > > > > > > > there's sufficient interest.
>> > > > > > > >
>> > > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <
>> sanand@apache.org>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Sounds great Joy!
>> > > > > > > > >
>> > > > > > > > > I've promoted you to *Event Organizer* on the Bay Area
>> Apache
>> > > > > Airflow
>> > > > > > > > > Meetup as per instructions/guidelines on
>> > > > > > > > > https://cwiki.apache.org/confl
>> uence/display/AIRFLOW/Meetups
>> > > > > > > > >
>> > > > > > > > > Go ahead and start setting up the meetup... we recommend 3
>> > > > speakers
>> > > > > > > with
>> > > > > > > > > 1-2 of them being external to the hosting company.
>> > > > > > > > >
>> > > > > > > > > Once the meetup date is setup, we can add it to
>> > > > > > > > >
>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
>> > > > > &
>> > > > > > > > tweet
>> > > > > > > > > it or share it over the dev list.
>> > > > > > > > > -s
>> > > > > > > > >
>> > > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com>
>> > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi folks,
>> > > > > > > > > >
>> > > > > > > > > > I'm Joy from WePay. At the last Airflow meetup in
>> December
>> > > > there
>> > > > > > was
>> > > > > > > a
>> > > > > > > > > > demand for hosting a future meetup to cover topics on
>> > Airflow
>> > > > > > > > > integrations
>> > > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS
>> > > hooks/operators,
>> > > > > best
>> > > > > > > > > > practices on running Airflow in the cloud, managed
>> Airflow,
>> > > > etc.)
>> > > > > > > > > >
>> > > > > > > > > > If there is enough interest, WePay can host the next
>> event
>> > > > > > sometimes
>> > > > > > > in
>> > > > > > > > > > March/April. And if you would like to give a talk or
>> have
>> > > > another
>> > > > > > > topic
>> > > > > > > > > > covered, feel free to suggest them as well.
>> > > > > > > > > >
>> > > > > > > > > > Cheers,
>> > > > > > > > > > Joy
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Q1 Airflow Bay Area Meetup
Posted by Sumit Maheshwari <ms...@apache.org>.
>
> We can present our internal tool for doing data quality checks for ETL's
> in Qubole.
> Iris is an Airflow Operator that currently has built-in functions for count
> validation, data-profile validation and duplicate data validation.
That would be great Devjyoti.
folks,
I would really love to attend this meetup and meet the Airflow community.
I'll be attending dataplatforms <https://www.dataplatforms.com/>conference
between 11-13th Apr and would be in the bay area before and after that.
On the Airflow meetup page, I see that it's scheduled on 11th Apr, so if
possible can we move it to 13th or some other day, it would be great.
Thanks,
Sumit
On Wed, Jan 31, 2018 at 1:52 PM, Devjyoti Patra <de...@qubole.com>
wrote:
> We can present our internal tool for doing data quality checks for ETL's in
> Qubole.
> Iris is an Airflow Operator that currently has built-in functions for
> count validation,
> data-profile validation and duplicate data validation.
>
> On Wed, Jan 31, 2018 at 3:10 AM, George Leslie-Waksman <
> george@cloverhealth.com.invalid> wrote:
>
> > If you're still looking for speakers, I have a handful of things I could
> > pull together to talk about:
> >
> > - the interplay between pools, queues, parallelism, dag concurrency,
> and
> > other job scheduling levers
> > - layering additional instrumentation on top of Airflow operators
> > - integrity testing DAGs in CI before they hit production / unit
> testing
> > tasks
> > - using Airflow metadata to track systemic execution behavior
> > - how I would have used Airflow if I'd known then what I know now
> >
> >
> > On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <an...@gmail.com>
> > wrote:
> >
> > > +1 on being able to watch a recording, based in Dublin, Ireland but
> also
> > > very interested in the cloud side of airflow.
> > >
> > > On Thu, 11 Jan 2018, 17:59 Joy Gao, <jo...@wepay.com> wrote:
> > >
> > > > Great to see that there's a lot of interests!
> > > >
> > > > I will go ahead and work on organizing this event. Details TBD.
> > > >
> > > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <
> > llorenz@industrydive.com>
> > > > wrote:
> > > >
> > > > > I would love to see videos from this as well, as we're east coast.
> We
> > > > host
> > > > > our airflow installation in GCP using GKE, and would love to
> compare
> > > > notes.
> > > > >
> > > > > Laura
> > > > >
> > > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bd...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > If it coincides with qcon.ai (april 10-11) or is close to its
> > dates
> > > > > then I
> > > > > > can join.
> > > > > >
> > > > > > B.
> > > > > >
> > > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <
> > vananth22@gmail.com
> > > >:
> > > > > >
> > > > > > I can give a talk about all the hacks we did to scale Airflow
> Local
> > > > > > Executor and improve the data pipeline on-call experience at
> Slack
> > if
> > > > > folks
> > > > > > are interested.
> > > > > >
> > > > > > Regards,
> > > > > > Ananth.P,
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 8 January 2018 at 15:51, George Leslie-Waksman <
> > > > > > george@cloverhealth.com.invalid> wrote:
> > > > > >
> > > > > > > +1 and would love to hear about other folks running Airflow in
> > GCP.
> > > > > > >
> > > > > > > Would strongly prefer March (but I'm just one person)
> > > > > > >
> > > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu
> <fenglu@google.com.invalid
> > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1 Joy!
> > > > > > > >
> > > > > > > > Would prefer April if possible, I can talk about running
> > Airflow
> > > in
> > > > > GCP
> > > > > > > if
> > > > > > > > there's sufficient interest.
> > > > > > > >
> > > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <
> sanand@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Sounds great Joy!
> > > > > > > > >
> > > > > > > > > I've promoted you to *Event Organizer* on the Bay Area
> Apache
> > > > > Airflow
> > > > > > > > > Meetup as per instructions/guidelines on
> > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/
> Meetups
> > > > > > > > >
> > > > > > > > > Go ahead and start setting up the meetup... we recommend 3
> > > > speakers
> > > > > > > with
> > > > > > > > > 1-2 of them being external to the hosting company.
> > > > > > > > >
> > > > > > > > > Once the meetup date is setup, we can add it to
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
> > > > > &
> > > > > > > > tweet
> > > > > > > > > it or share it over the dev list.
> > > > > > > > > -s
> > > > > > > > >
> > > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi folks,
> > > > > > > > > >
> > > > > > > > > > I'm Joy from WePay. At the last Airflow meetup in
> December
> > > > there
> > > > > > was
> > > > > > > a
> > > > > > > > > > demand for hosting a future meetup to cover topics on
> > Airflow
> > > > > > > > > integrations
> > > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS
> > > hooks/operators,
> > > > > best
> > > > > > > > > > practices on running Airflow in the cloud, managed
> Airflow,
> > > > etc.)
> > > > > > > > > >
> > > > > > > > > > If there is enough interest, WePay can host the next
> event
> > > > > > sometimes
> > > > > > > in
> > > > > > > > > > March/April. And if you would like to give a talk or have
> > > > another
> > > > > > > topic
> > > > > > > > > > covered, feel free to suggest them as well.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Joy
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Devjyoti Patra <de...@qubole.com>.
We can present our internal tool for doing data quality checks for ETL's in
Qubole.
Iris is an Airflow Operator that currently has built-in functions for
count validation,
data-profile validation and duplicate data validation.
On Wed, Jan 31, 2018 at 3:10 AM, George Leslie-Waksman <
george@cloverhealth.com.invalid> wrote:
> If you're still looking for speakers, I have a handful of things I could
> pull together to talk about:
>
> - the interplay between pools, queues, parallelism, dag concurrency, and
> other job scheduling levers
> - layering additional instrumentation on top of Airflow operators
> - integrity testing DAGs in CI before they hit production / unit testing
> tasks
> - using Airflow metadata to track systemic execution behavior
> - how I would have used Airflow if I'd known then what I know now
>
>
> On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <an...@gmail.com>
> wrote:
>
> > +1 on being able to watch a recording, based in Dublin, Ireland but also
> > very interested in the cloud side of airflow.
> >
> > On Thu, 11 Jan 2018, 17:59 Joy Gao, <jo...@wepay.com> wrote:
> >
> > > Great to see that there's a lot of interests!
> > >
> > > I will go ahead and work on organizing this event. Details TBD.
> > >
> > > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <
> llorenz@industrydive.com>
> > > wrote:
> > >
> > > > I would love to see videos from this as well, as we're east coast. We
> > > host
> > > > our airflow installation in GCP using GKE, and would love to compare
> > > notes.
> > > >
> > > > Laura
> > > >
> > > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > > >
> > > > > If it coincides with qcon.ai (april 10-11) or is close to its
> dates
> > > > then I
> > > > > can join.
> > > > >
> > > > > B.
> > > > >
> > > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <
> vananth22@gmail.com
> > >:
> > > > >
> > > > > I can give a talk about all the hacks we did to scale Airflow Local
> > > > > Executor and improve the data pipeline on-call experience at Slack
> if
> > > > folks
> > > > > are interested.
> > > > >
> > > > > Regards,
> > > > > Ananth.P,
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 8 January 2018 at 15:51, George Leslie-Waksman <
> > > > > george@cloverhealth.com.invalid> wrote:
> > > > >
> > > > > > +1 and would love to hear about other folks running Airflow in
> GCP.
> > > > > >
> > > > > > Would strongly prefer March (but I'm just one person)
> > > > > >
> > > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fenglu@google.com.invalid
> >
> > > > > wrote:
> > > > > >
> > > > > > > +1 Joy!
> > > > > > >
> > > > > > > Would prefer April if possible, I can talk about running
> Airflow
> > in
> > > > GCP
> > > > > > if
> > > > > > > there's sufficient interest.
> > > > > > >
> > > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Sounds great Joy!
> > > > > > > >
> > > > > > > > I've promoted you to *Event Organizer* on the Bay Area Apache
> > > > Airflow
> > > > > > > > Meetup as per instructions/guidelines on
> > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > > > > > > >
> > > > > > > > Go ahead and start setting up the meetup... we recommend 3
> > > speakers
> > > > > > with
> > > > > > > > 1-2 of them being external to the hosting company.
> > > > > > > >
> > > > > > > > Once the meetup date is setup, we can add it to
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
> > > > &
> > > > > > > tweet
> > > > > > > > it or share it over the dev list.
> > > > > > > > -s
> > > > > > > >
> > > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com>
> > wrote:
> > > > > > > >
> > > > > > > > > Hi folks,
> > > > > > > > >
> > > > > > > > > I'm Joy from WePay. At the last Airflow meetup in December
> > > there
> > > > > was
> > > > > > a
> > > > > > > > > demand for hosting a future meetup to cover topics on
> Airflow
> > > > > > > > integrations
> > > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS
> > hooks/operators,
> > > > best
> > > > > > > > > practices on running Airflow in the cloud, managed Airflow,
> > > etc.)
> > > > > > > > >
> > > > > > > > > If there is enough interest, WePay can host the next event
> > > > > sometimes
> > > > > > in
> > > > > > > > > March/April. And if you would like to give a talk or have
> > > another
> > > > > > topic
> > > > > > > > > covered, feel free to suggest them as well.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Joy
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by George Leslie-Waksman <ge...@cloverhealth.com.INVALID>.
If you're still looking for speakers, I have a handful of things I could
pull together to talk about:
- the interplay between pools, queues, parallelism, dag concurrency, and
other job scheduling levers
- layering additional instrumentation on top of Airflow operators
- integrity testing DAGs in CI before they hit production / unit testing
tasks
- using Airflow metadata to track systemic execution behavior
- how I would have used Airflow if I'd known then what I know now
On Thu, Jan 11, 2018 at 12:18 PM Andrew Maguire <an...@gmail.com>
wrote:
> +1 on being able to watch a recording, based in Dublin, Ireland but also
> very interested in the cloud side of airflow.
>
> On Thu, 11 Jan 2018, 17:59 Joy Gao, <jo...@wepay.com> wrote:
>
> > Great to see that there's a lot of interests!
> >
> > I will go ahead and work on organizing this event. Details TBD.
> >
> > On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <ll...@industrydive.com>
> > wrote:
> >
> > > I would love to see videos from this as well, as we're east coast. We
> > host
> > > our airflow installation in GCP using GKE, and would love to compare
> > notes.
> > >
> > > Laura
> > >
> > > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > > > If it coincides with qcon.ai (april 10-11) or is close to its dates
> > > then I
> > > > can join.
> > > >
> > > > B.
> > > >
> > > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <vananth22@gmail.com
> >:
> > > >
> > > > I can give a talk about all the hacks we did to scale Airflow Local
> > > > Executor and improve the data pipeline on-call experience at Slack if
> > > folks
> > > > are interested.
> > > >
> > > > Regards,
> > > > Ananth.P,
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 8 January 2018 at 15:51, George Leslie-Waksman <
> > > > george@cloverhealth.com.invalid> wrote:
> > > >
> > > > > +1 and would love to hear about other folks running Airflow in GCP.
> > > > >
> > > > > Would strongly prefer March (but I'm just one person)
> > > > >
> > > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid>
> > > > wrote:
> > > > >
> > > > > > +1 Joy!
> > > > > >
> > > > > > Would prefer April if possible, I can talk about running Airflow
> in
> > > GCP
> > > > > if
> > > > > > there's sufficient interest.
> > > > > >
> > > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Sounds great Joy!
> > > > > > >
> > > > > > > I've promoted you to *Event Organizer* on the Bay Area Apache
> > > Airflow
> > > > > > > Meetup as per instructions/guidelines on
> > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > > > > > >
> > > > > > > Go ahead and start setting up the meetup... we recommend 3
> > speakers
> > > > > with
> > > > > > > 1-2 of them being external to the hosting company.
> > > > > > >
> > > > > > > Once the meetup date is setup, we can add it to
> > > > > > >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
> > > &
> > > > > > tweet
> > > > > > > it or share it over the dev list.
> > > > > > > -s
> > > > > > >
> > > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com>
> wrote:
> > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > I'm Joy from WePay. At the last Airflow meetup in December
> > there
> > > > was
> > > > > a
> > > > > > > > demand for hosting a future meetup to cover topics on Airflow
> > > > > > > integrations
> > > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS
> hooks/operators,
> > > best
> > > > > > > > practices on running Airflow in the cloud, managed Airflow,
> > etc.)
> > > > > > > >
> > > > > > > > If there is enough interest, WePay can host the next event
> > > > sometimes
> > > > > in
> > > > > > > > March/April. And if you would like to give a talk or have
> > another
> > > > > topic
> > > > > > > > covered, feel free to suggest them as well.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Joy
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Andrew Maguire <an...@gmail.com>.
+1 on being able to watch a recording, based in Dublin, Ireland but also
very interested in the cloud side of airflow.
On Thu, 11 Jan 2018, 17:59 Joy Gao, <jo...@wepay.com> wrote:
> Great to see that there's a lot of interests!
>
> I will go ahead and work on organizing this event. Details TBD.
>
> On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <ll...@industrydive.com>
> wrote:
>
> > I would love to see videos from this as well, as we're east coast. We
> host
> > our airflow installation in GCP using GKE, and would love to compare
> notes.
> >
> > Laura
> >
> > On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> > > If it coincides with qcon.ai (april 10-11) or is close to its dates
> > then I
> > > can join.
> > >
> > > B.
> > >
> > > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <va...@gmail.com>:
> > >
> > > I can give a talk about all the hacks we did to scale Airflow Local
> > > Executor and improve the data pipeline on-call experience at Slack if
> > folks
> > > are interested.
> > >
> > > Regards,
> > > Ananth.P,
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 8 January 2018 at 15:51, George Leslie-Waksman <
> > > george@cloverhealth.com.invalid> wrote:
> > >
> > > > +1 and would love to hear about other folks running Airflow in GCP.
> > > >
> > > > Would strongly prefer March (but I'm just one person)
> > > >
> > > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid>
> > > wrote:
> > > >
> > > > > +1 Joy!
> > > > >
> > > > > Would prefer April if possible, I can talk about running Airflow in
> > GCP
> > > > if
> > > > > there's sufficient interest.
> > > > >
> > > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org>
> > wrote:
> > > > >
> > > > > > Sounds great Joy!
> > > > > >
> > > > > > I've promoted you to *Event Organizer* on the Bay Area Apache
> > Airflow
> > > > > > Meetup as per instructions/guidelines on
> > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > > > > >
> > > > > > Go ahead and start setting up the meetup... we recommend 3
> speakers
> > > > with
> > > > > > 1-2 of them being external to the hosting company.
> > > > > >
> > > > > > Once the meetup date is setup, we can add it to
> > > > > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
> > &
> > > > > tweet
> > > > > > it or share it over the dev list.
> > > > > > -s
> > > > > >
> > > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > I'm Joy from WePay. At the last Airflow meetup in December
> there
> > > was
> > > > a
> > > > > > > demand for hosting a future meetup to cover topics on Airflow
> > > > > > integrations
> > > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators,
> > best
> > > > > > > practices on running Airflow in the cloud, managed Airflow,
> etc.)
> > > > > > >
> > > > > > > If there is enough interest, WePay can host the next event
> > > sometimes
> > > > in
> > > > > > > March/April. And if you would like to give a talk or have
> another
> > > > topic
> > > > > > > covered, feel free to suggest them as well.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Joy
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Joy Gao <jo...@wepay.com>.
Great to see that there's a lot of interests!
I will go ahead and work on organizing this event. Details TBD.
On Wed, Jan 10, 2018 at 8:03 AM, Laura Lorenz <ll...@industrydive.com>
wrote:
> I would love to see videos from this as well, as we're east coast. We host
> our airflow installation in GCP using GKE, and would love to compare notes.
>
> Laura
>
> On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > If it coincides with qcon.ai (april 10-11) or is close to its dates
> then I
> > can join.
> >
> > B.
> >
> > Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <va...@gmail.com>:
> >
> > I can give a talk about all the hacks we did to scale Airflow Local
> > Executor and improve the data pipeline on-call experience at Slack if
> folks
> > are interested.
> >
> > Regards,
> > Ananth.P,
> >
> >
> >
> >
> >
> >
> > On 8 January 2018 at 15:51, George Leslie-Waksman <
> > george@cloverhealth.com.invalid> wrote:
> >
> > > +1 and would love to hear about other folks running Airflow in GCP.
> > >
> > > Would strongly prefer March (but I'm just one person)
> > >
> > > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid>
> > wrote:
> > >
> > > > +1 Joy!
> > > >
> > > > Would prefer April if possible, I can talk about running Airflow in
> GCP
> > > if
> > > > there's sufficient interest.
> > > >
> > > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org>
> wrote:
> > > >
> > > > > Sounds great Joy!
> > > > >
> > > > > I've promoted you to *Event Organizer* on the Bay Area Apache
> Airflow
> > > > > Meetup as per instructions/guidelines on
> > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > > > >
> > > > > Go ahead and start setting up the meetup... we recommend 3 speakers
> > > with
> > > > > 1-2 of them being external to the hosting company.
> > > > >
> > > > > Once the meetup date is setup, we can add it to
> > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
> &
> > > > tweet
> > > > > it or share it over the dev list.
> > > > > -s
> > > > >
> > > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > I'm Joy from WePay. At the last Airflow meetup in December there
> > was
> > > a
> > > > > > demand for hosting a future meetup to cover topics on Airflow
> > > > > integrations
> > > > > > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators,
> best
> > > > > > practices on running Airflow in the cloud, managed Airflow, etc.)
> > > > > >
> > > > > > If there is enough interest, WePay can host the next event
> > sometimes
> > > in
> > > > > > March/April. And if you would like to give a talk or have another
> > > topic
> > > > > > covered, feel free to suggest them as well.
> > > > > >
> > > > > > Cheers,
> > > > > > Joy
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Laura Lorenz <ll...@industrydive.com>.
I would love to see videos from this as well, as we're east coast. We host
our airflow installation in GCP using GKE, and would love to compare notes.
Laura
On Tue, Jan 9, 2018 at 4:57 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> If it coincides with qcon.ai (april 10-11) or is close to its dates then I
> can join.
>
> B.
>
> Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <va...@gmail.com>:
>
> I can give a talk about all the hacks we did to scale Airflow Local
> Executor and improve the data pipeline on-call experience at Slack if folks
> are interested.
>
> Regards,
> Ananth.P,
>
>
>
>
>
>
> On 8 January 2018 at 15:51, George Leslie-Waksman <
> george@cloverhealth.com.invalid> wrote:
>
> > +1 and would love to hear about other folks running Airflow in GCP.
> >
> > Would strongly prefer March (but I'm just one person)
> >
> > On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid>
> wrote:
> >
> > > +1 Joy!
> > >
> > > Would prefer April if possible, I can talk about running Airflow in GCP
> > if
> > > there's sufficient interest.
> > >
> > > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org> wrote:
> > >
> > > > Sounds great Joy!
> > > >
> > > > I've promoted you to *Event Organizer* on the Bay Area Apache Airflow
> > > > Meetup as per instructions/guidelines on
> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > > >
> > > > Go ahead and start setting up the meetup... we recommend 3 speakers
> > with
> > > > 1-2 of them being external to the hosting company.
> > > >
> > > > Once the meetup date is setup, we can add it to
> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
> > > tweet
> > > > it or share it over the dev list.
> > > > -s
> > > >
> > > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I'm Joy from WePay. At the last Airflow meetup in December there
> was
> > a
> > > > > demand for hosting a future meetup to cover topics on Airflow
> > > > integrations
> > > > > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
> > > > > practices on running Airflow in the cloud, managed Airflow, etc.)
> > > > >
> > > > > If there is enough interest, WePay can host the next event
> sometimes
> > in
> > > > > March/April. And if you would like to give a talk or have another
> > topic
> > > > > covered, feel free to suggest them as well.
> > > > >
> > > > > Cheers,
> > > > > Joy
> > > > >
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Bolke de Bruin <bd...@gmail.com>.
If it coincides with qcon.ai (april 10-11) or is close to its dates then I
can join.
B.
Op 9 jan. 2018 5:36 a.m. schreef "Ananth Durai" <va...@gmail.com>:
I can give a talk about all the hacks we did to scale Airflow Local
Executor and improve the data pipeline on-call experience at Slack if folks
are interested.
Regards,
Ananth.P,
On 8 January 2018 at 15:51, George Leslie-Waksman <
george@cloverhealth.com.invalid> wrote:
> +1 and would love to hear about other folks running Airflow in GCP.
>
> Would strongly prefer March (but I'm just one person)
>
> On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid> wrote:
>
> > +1 Joy!
> >
> > Would prefer April if possible, I can talk about running Airflow in GCP
> if
> > there's sufficient interest.
> >
> > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org> wrote:
> >
> > > Sounds great Joy!
> > >
> > > I've promoted you to *Event Organizer* on the Bay Area Apache Airflow
> > > Meetup as per instructions/guidelines on
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > >
> > > Go ahead and start setting up the meetup... we recommend 3 speakers
> with
> > > 1-2 of them being external to the hosting company.
> > >
> > > Once the meetup date is setup, we can add it to
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
> > tweet
> > > it or share it over the dev list.
> > > -s
> > >
> > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I'm Joy from WePay. At the last Airflow meetup in December there was
> a
> > > > demand for hosting a future meetup to cover topics on Airflow
> > > integrations
> > > > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
> > > > practices on running Airflow in the cloud, managed Airflow, etc.)
> > > >
> > > > If there is enough interest, WePay can host the next event sometimes
> in
> > > > March/April. And if you would like to give a talk or have another
> topic
> > > > covered, feel free to suggest them as well.
> > > >
> > > > Cheers,
> > > > Joy
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Ananth Durai <va...@gmail.com>.
I can give a talk about all the hacks we did to scale Airflow Local
Executor and improve the data pipeline on-call experience at Slack if folks
are interested.
Regards,
Ananth.P,
On 8 January 2018 at 15:51, George Leslie-Waksman <
george@cloverhealth.com.invalid> wrote:
> +1 and would love to hear about other folks running Airflow in GCP.
>
> Would strongly prefer March (but I'm just one person)
>
> On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid> wrote:
>
> > +1 Joy!
> >
> > Would prefer April if possible, I can talk about running Airflow in GCP
> if
> > there's sufficient interest.
> >
> > On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org> wrote:
> >
> > > Sounds great Joy!
> > >
> > > I've promoted you to *Event Organizer* on the Bay Area Apache Airflow
> > > Meetup as per instructions/guidelines on
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> > >
> > > Go ahead and start setting up the meetup... we recommend 3 speakers
> with
> > > 1-2 of them being external to the hosting company.
> > >
> > > Once the meetup date is setup, we can add it to
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
> > tweet
> > > it or share it over the dev list.
> > > -s
> > >
> > > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I'm Joy from WePay. At the last Airflow meetup in December there was
> a
> > > > demand for hosting a future meetup to cover topics on Airflow
> > > integrations
> > > > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
> > > > practices on running Airflow in the cloud, managed Airflow, etc.)
> > > >
> > > > If there is enough interest, WePay can host the next event sometimes
> in
> > > > March/April. And if you would like to give a talk or have another
> topic
> > > > covered, feel free to suggest them as well.
> > > >
> > > > Cheers,
> > > > Joy
> > > >
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by George Leslie-Waksman <ge...@cloverhealth.com.INVALID>.
+1 and would love to hear about other folks running Airflow in GCP.
Would strongly prefer March (but I'm just one person)
On Mon, Jan 8, 2018 at 2:30 PM Feng Lu <fe...@google.com.invalid> wrote:
> +1 Joy!
>
> Would prefer April if possible, I can talk about running Airflow in GCP if
> there's sufficient interest.
>
> On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org> wrote:
>
> > Sounds great Joy!
> >
> > I've promoted you to *Event Organizer* on the Bay Area Apache Airflow
> > Meetup as per instructions/guidelines on
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> >
> > Go ahead and start setting up the meetup... we recommend 3 speakers with
> > 1-2 of them being external to the hosting company.
> >
> > Once the meetup date is setup, we can add it to
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
> tweet
> > it or share it over the dev list.
> > -s
> >
> > On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> > > Hi folks,
> > >
> > > I'm Joy from WePay. At the last Airflow meetup in December there was a
> > > demand for hosting a future meetup to cover topics on Airflow
> > integrations
> > > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
> > > practices on running Airflow in the cloud, managed Airflow, etc.)
> > >
> > > If there is enough interest, WePay can host the next event sometimes in
> > > March/April. And if you would like to give a talk or have another topic
> > > covered, feel free to suggest them as well.
> > >
> > > Cheers,
> > > Joy
> > >
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Feng Lu <fe...@google.com.INVALID>.
+1 Joy!
Would prefer April if possible, I can talk about running Airflow in GCP if
there's sufficient interest.
On Fri, Jan 5, 2018 at 12:59 PM, Sid Anand <sa...@apache.org> wrote:
> Sounds great Joy!
>
> I've promoted you to *Event Organizer* on the Bay Area Apache Airflow
> Meetup as per instructions/guidelines on
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
>
> Go ahead and start setting up the meetup... we recommend 3 speakers with
> 1-2 of them being external to the hosting company.
>
> Once the meetup date is setup, we can add it to
> https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements & tweet
> it or share it over the dev list.
> -s
>
> On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
>
> > Hi folks,
> >
> > I'm Joy from WePay. At the last Airflow meetup in December there was a
> > demand for hosting a future meetup to cover topics on Airflow
> integrations
> > with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
> > practices on running Airflow in the cloud, managed Airflow, etc.)
> >
> > If there is enough interest, WePay can host the next event sometimes in
> > March/April. And if you would like to give a talk or have another topic
> > covered, feel free to suggest them as well.
> >
> > Cheers,
> > Joy
> >
>
Re: Q1 Airflow Bay Area Meetup
Posted by Sid Anand <sa...@apache.org>.
Sounds great Joy!
I've promoted you to *Event Organizer* on the Bay Area Apache Airflow
Meetup as per instructions/guidelines on
https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
Go ahead and start setting up the meetup... we recommend 3 speakers with
1-2 of them being external to the hosting company.
Once the meetup date is setup, we can add it to
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements & tweet
it or share it over the dev list.
-s
On Fri, Jan 5, 2018 at 12:10 PM, Joy Gao <jo...@wepay.com> wrote:
> Hi folks,
>
> I'm Joy from WePay. At the last Airflow meetup in December there was a
> demand for hosting a future meetup to cover topics on Airflow integrations
> with GCP/AWS (for example, CI/CD with GCP/AWS hooks/operators, best
> practices on running Airflow in the cloud, managed Airflow, etc.)
>
> If there is enough interest, WePay can host the next event sometimes in
> March/April. And if you would like to give a talk or have another topic
> covered, feel free to suggest them as well.
>
> Cheers,
> Joy
>