You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ryan Abernathey <ry...@gmail.com> on 2022/06/08 14:22:41 UTC
Join a meeting to help coordinate implementing a Dask Runner for Beam
Dear Beamer,
Thank you for all of your work on this amazing project. I am new to Beam
and am quite excited about its potential to help with some data processing
challenges in my field of climate science.
Our community is interested in running Beam on Dask Distributed clusters,
which we already know how to deploy. This has been discussed at
https://issues.apache.org/jira/browse/BEAM-5336 and
https://github.com/apache/beam/issues/18962. It seems technically feasible.
We are trying to organize a meeting next week to kickstart and coordinate
this effort. It would be great if we could entrain some Beam maintainers
into this meeting. If you have interest in this topic and are available
next week, please share your availability here -
https://www.when2meet.com/?15861604-jLnA4
Alternatively, if you have any guidance or suggestions you wish to provide
by email or GitHub discussion, we welcome your input.
Thanks again for your open source work.
Best,
Ryan Abernathey
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Posted by Brian Hulette via dev <de...@beam.apache.org>.
I wanted to share that Ryan gave a presentation about his (and Charles')
work on Pangeo Forge at Scipy 2022 (in Austin just before Beam Summit!),
with a couple mentions of their transition to Beam [1]. There were also a
couple of other talks about Pangeo [2,3] with some Beam/xarray-beam
references in there.
[1]
https://www.youtube.com/watch?v=sY20UpYCAEE&list=PLYx7XA2nY5Gde0WF1yswQw5InhmSNED8o&index=9
[2]
https://www.youtube.com/watch?v=7niNfs3ZpfQ&list=PLYx7XA2nY5Gfb0tQyezb4Gsf1nVsy86zt&index=2
[3]
https://www.youtube.com/watch?v=ftlgOESINvo&list=PLYx7XA2nY5Gfb0tQyezb4Gsf1nVsy86zt&index=3
On Tue, Jun 21, 2022 at 9:29 AM Ahmet Altay <al...@google.com> wrote:
> Were you able to meet? If yes, I would be very interested in a summary if
> someone would like to share that :)
>
> On Mon, Jun 13, 2022 at 9:16 AM Pablo Estrada <pa...@google.com> wrote:
>
>> Also added my availability... please do invite me as well : )
>> -P.
>>
>> On Mon, Jun 13, 2022 at 6:57 AM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> I would love to try to join any meetings if you add me. My calendar is
>>> too chaotic to be useful on the when2meet :-) but I can often move things
>>> around.
>>>
>>> Kenn
>>>
>>> On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette <bh...@google.com>
>>> wrote:
>>>
>>>> Thanks for reaching out, Ryan, this sounds really cool. I added my
>>>> availability to the calendar since I'm interested in this space, but I'm
>>>> not sure I can offer much help - I don't have any experience building a
>>>> runner, to date I've worked exclusively on the SDK side of Beam. So I hope
>>>> some other folks can join as well :)
>>>>
>>>> @Pablo Estrada <pa...@google.com> might have some useful insight -
>>>> he's been working on a spike to build a Ray runner.
>>>>
>>>>
>>>> On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>>
>>>>> This sounds like a great project. Unfortunately I wouldn't be able to
>>>>> meet next week, but would be happy to meet some other time and if that
>>>>> doesn't work answer questions over email, etc. Looking forward to a
>>>>> Dask runner.
>>>>>
>>>>> On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey
>>>>> <ry...@gmail.com> wrote:
>>>>> >
>>>>> > Dear Beamer,
>>>>> >
>>>>> > Thank you for all of your work on this amazing project. I am new to
>>>>> Beam and am quite excited about its potential to help with some data
>>>>> processing challenges in my field of climate science.
>>>>> >
>>>>> > Our community is interested in running Beam on Dask Distributed
>>>>> clusters, which we already know how to deploy. This has been discussed at
>>>>> https://issues.apache.org/jira/browse/BEAM-5336 and
>>>>> https://github.com/apache/beam/issues/18962. It seems technically
>>>>> feasible.
>>>>> >
>>>>> > We are trying to organize a meeting next week to kickstart and
>>>>> coordinate this effort. It would be great if we could entrain some Beam
>>>>> maintainers into this meeting. If you have interest in this topic and are
>>>>> available next week, please share your availability here -
>>>>> https://www.when2meet.com/?15861604-jLnA4
>>>>> >
>>>>> > Alternatively, if you have any guidance or suggestions you wish to
>>>>> provide by email or GitHub discussion, we welcome your input.
>>>>> >
>>>>> > Thanks again for your open source work.
>>>>> >
>>>>> > Best,
>>>>> > Ryan Abernathey
>>>>> >
>>>>>
>>>>
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Posted by Ahmet Altay <al...@google.com>.
Were you able to meet? If yes, I would be very interested in a summary if
someone would like to share that :)
On Mon, Jun 13, 2022 at 9:16 AM Pablo Estrada <pa...@google.com> wrote:
> Also added my availability... please do invite me as well : )
> -P.
>
> On Mon, Jun 13, 2022 at 6:57 AM Kenneth Knowles <ke...@apache.org> wrote:
>
>> I would love to try to join any meetings if you add me. My calendar is
>> too chaotic to be useful on the when2meet :-) but I can often move things
>> around.
>>
>> Kenn
>>
>> On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette <bh...@google.com> wrote:
>>
>>> Thanks for reaching out, Ryan, this sounds really cool. I added my
>>> availability to the calendar since I'm interested in this space, but I'm
>>> not sure I can offer much help - I don't have any experience building a
>>> runner, to date I've worked exclusively on the SDK side of Beam. So I hope
>>> some other folks can join as well :)
>>>
>>> @Pablo Estrada <pa...@google.com> might have some useful insight -
>>> he's been working on a spike to build a Ray runner.
>>>
>>>
>>> On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> This sounds like a great project. Unfortunately I wouldn't be able to
>>>> meet next week, but would be happy to meet some other time and if that
>>>> doesn't work answer questions over email, etc. Looking forward to a
>>>> Dask runner.
>>>>
>>>> On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey
>>>> <ry...@gmail.com> wrote:
>>>> >
>>>> > Dear Beamer,
>>>> >
>>>> > Thank you for all of your work on this amazing project. I am new to
>>>> Beam and am quite excited about its potential to help with some data
>>>> processing challenges in my field of climate science.
>>>> >
>>>> > Our community is interested in running Beam on Dask Distributed
>>>> clusters, which we already know how to deploy. This has been discussed at
>>>> https://issues.apache.org/jira/browse/BEAM-5336 and
>>>> https://github.com/apache/beam/issues/18962. It seems technically
>>>> feasible.
>>>> >
>>>> > We are trying to organize a meeting next week to kickstart and
>>>> coordinate this effort. It would be great if we could entrain some Beam
>>>> maintainers into this meeting. If you have interest in this topic and are
>>>> available next week, please share your availability here -
>>>> https://www.when2meet.com/?15861604-jLnA4
>>>> >
>>>> > Alternatively, if you have any guidance or suggestions you wish to
>>>> provide by email or GitHub discussion, we welcome your input.
>>>> >
>>>> > Thanks again for your open source work.
>>>> >
>>>> > Best,
>>>> > Ryan Abernathey
>>>> >
>>>>
>>>
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Posted by Pablo Estrada <pa...@google.com>.
Also added my availability... please do invite me as well : )
-P.
On Mon, Jun 13, 2022 at 6:57 AM Kenneth Knowles <ke...@apache.org> wrote:
> I would love to try to join any meetings if you add me. My calendar is too
> chaotic to be useful on the when2meet :-) but I can often move things
> around.
>
> Kenn
>
> On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette <bh...@google.com> wrote:
>
>> Thanks for reaching out, Ryan, this sounds really cool. I added my
>> availability to the calendar since I'm interested in this space, but I'm
>> not sure I can offer much help - I don't have any experience building a
>> runner, to date I've worked exclusively on the SDK side of Beam. So I hope
>> some other folks can join as well :)
>>
>> @Pablo Estrada <pa...@google.com> might have some useful insight -
>> he's been working on a spike to build a Ray runner.
>>
>>
>> On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> This sounds like a great project. Unfortunately I wouldn't be able to
>>> meet next week, but would be happy to meet some other time and if that
>>> doesn't work answer questions over email, etc. Looking forward to a
>>> Dask runner.
>>>
>>> On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey
>>> <ry...@gmail.com> wrote:
>>> >
>>> > Dear Beamer,
>>> >
>>> > Thank you for all of your work on this amazing project. I am new to
>>> Beam and am quite excited about its potential to help with some data
>>> processing challenges in my field of climate science.
>>> >
>>> > Our community is interested in running Beam on Dask Distributed
>>> clusters, which we already know how to deploy. This has been discussed at
>>> https://issues.apache.org/jira/browse/BEAM-5336 and
>>> https://github.com/apache/beam/issues/18962. It seems technically
>>> feasible.
>>> >
>>> > We are trying to organize a meeting next week to kickstart and
>>> coordinate this effort. It would be great if we could entrain some Beam
>>> maintainers into this meeting. If you have interest in this topic and are
>>> available next week, please share your availability here -
>>> https://www.when2meet.com/?15861604-jLnA4
>>> >
>>> > Alternatively, if you have any guidance or suggestions you wish to
>>> provide by email or GitHub discussion, we welcome your input.
>>> >
>>> > Thanks again for your open source work.
>>> >
>>> > Best,
>>> > Ryan Abernathey
>>> >
>>>
>>
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Posted by Kenneth Knowles <ke...@apache.org>.
I would love to try to join any meetings if you add me. My calendar is too
chaotic to be useful on the when2meet :-) but I can often move things
around.
Kenn
On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette <bh...@google.com> wrote:
> Thanks for reaching out, Ryan, this sounds really cool. I added my
> availability to the calendar since I'm interested in this space, but I'm
> not sure I can offer much help - I don't have any experience building a
> runner, to date I've worked exclusively on the SDK side of Beam. So I hope
> some other folks can join as well :)
>
> @Pablo Estrada <pa...@google.com> might have some useful insight - he's
> been working on a spike to build a Ray runner.
>
>
> On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> This sounds like a great project. Unfortunately I wouldn't be able to
>> meet next week, but would be happy to meet some other time and if that
>> doesn't work answer questions over email, etc. Looking forward to a
>> Dask runner.
>>
>> On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey
>> <ry...@gmail.com> wrote:
>> >
>> > Dear Beamer,
>> >
>> > Thank you for all of your work on this amazing project. I am new to
>> Beam and am quite excited about its potential to help with some data
>> processing challenges in my field of climate science.
>> >
>> > Our community is interested in running Beam on Dask Distributed
>> clusters, which we already know how to deploy. This has been discussed at
>> https://issues.apache.org/jira/browse/BEAM-5336 and
>> https://github.com/apache/beam/issues/18962. It seems technically
>> feasible.
>> >
>> > We are trying to organize a meeting next week to kickstart and
>> coordinate this effort. It would be great if we could entrain some Beam
>> maintainers into this meeting. If you have interest in this topic and are
>> available next week, please share your availability here -
>> https://www.when2meet.com/?15861604-jLnA4
>> >
>> > Alternatively, if you have any guidance or suggestions you wish to
>> provide by email or GitHub discussion, we welcome your input.
>> >
>> > Thanks again for your open source work.
>> >
>> > Best,
>> > Ryan Abernathey
>> >
>>
>
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Posted by Brian Hulette <bh...@google.com>.
Thanks for reaching out, Ryan, this sounds really cool. I added my
availability to the calendar since I'm interested in this space, but I'm
not sure I can offer much help - I don't have any experience building a
runner, to date I've worked exclusively on the SDK side of Beam. So I hope
some other folks can join as well :)
@Pablo Estrada <pa...@google.com> might have some useful insight - he's
been working on a spike to build a Ray runner.
On Wed, Jun 8, 2022 at 12:53 PM Robert Bradshaw <ro...@google.com> wrote:
> This sounds like a great project. Unfortunately I wouldn't be able to
> meet next week, but would be happy to meet some other time and if that
> doesn't work answer questions over email, etc. Looking forward to a
> Dask runner.
>
> On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey
> <ry...@gmail.com> wrote:
> >
> > Dear Beamer,
> >
> > Thank you for all of your work on this amazing project. I am new to Beam
> and am quite excited about its potential to help with some data processing
> challenges in my field of climate science.
> >
> > Our community is interested in running Beam on Dask Distributed
> clusters, which we already know how to deploy. This has been discussed at
> https://issues.apache.org/jira/browse/BEAM-5336 and
> https://github.com/apache/beam/issues/18962. It seems technically
> feasible.
> >
> > We are trying to organize a meeting next week to kickstart and
> coordinate this effort. It would be great if we could entrain some Beam
> maintainers into this meeting. If you have interest in this topic and are
> available next week, please share your availability here -
> https://www.when2meet.com/?15861604-jLnA4
> >
> > Alternatively, if you have any guidance or suggestions you wish to
> provide by email or GitHub discussion, we welcome your input.
> >
> > Thanks again for your open source work.
> >
> > Best,
> > Ryan Abernathey
> >
>
Re: Join a meeting to help coordinate implementing a Dask Runner for Beam
Posted by Robert Bradshaw <ro...@google.com>.
This sounds like a great project. Unfortunately I wouldn't be able to
meet next week, but would be happy to meet some other time and if that
doesn't work answer questions over email, etc. Looking forward to a
Dask runner.
On Wed, Jun 8, 2022 at 9:04 AM Ryan Abernathey
<ry...@gmail.com> wrote:
>
> Dear Beamer,
>
> Thank you for all of your work on this amazing project. I am new to Beam and am quite excited about its potential to help with some data processing challenges in my field of climate science.
>
> Our community is interested in running Beam on Dask Distributed clusters, which we already know how to deploy. This has been discussed at https://issues.apache.org/jira/browse/BEAM-5336 and https://github.com/apache/beam/issues/18962. It seems technically feasible.
>
> We are trying to organize a meeting next week to kickstart and coordinate this effort. It would be great if we could entrain some Beam maintainers into this meeting. If you have interest in this topic and are available next week, please share your availability here - https://www.when2meet.com/?15861604-jLnA4
>
> Alternatively, if you have any guidance or suggestions you wish to provide by email or GitHub discussion, we welcome your input.
>
> Thanks again for your open source work.
>
> Best,
> Ryan Abernathey
>