You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alan Myrvold <am...@google.com> on 2017/12/08 00:38:47 UTC
Introduction + interest in helping Beam builds, tests, and releases
Hi, I'm Alan.
I've been working with Google Cloud engineering productivity, and I'm keen
on improving the Beam release process and build/test infrastructure.
I will be first looking into scripting some of the release validation steps
for the nightly java snapshot releases, but hope to learn and improve the
whole development and testing experience for Beam.
Look forward to working with everyone!
Alan Myrvold
Re: Introduction + interest in helping Beam builds, tests, and
releases
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Alan,
it sounds great ! Welcome !
Please, don't hesitate to ping me if need (I think I know the release process
pretty well ;) and especially, if we can check together if the changes you may
do are compliant with Apache rules).
Looking forward to work with you !
Regards
JB
On 12/08/2017 01:38 AM, Alan Myrvold wrote:
> Hi, I'm Alan.
>
> I've been working with Google Cloud engineering productivity, and I'm keen on
> improving the Beam release process and build/test infrastructure.
>
> I will be first looking into scripting some of the release validation steps for
> the nightly java snapshot releases, but hope to learn and improve the whole
> development and testing experience for Beam.
>
> Look forward to working with everyone!
>
> Alan Myrvold
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Introduction + interest in helping Beam builds, tests, and releases
Posted by Reuven Lax <re...@google.com>.
Welcome Alan,
It's great to hear you'll be helping automate the release validation! This
will help ensure future releases happen faster, as we'll know each night
whether a release is likely to validate (instead of waiting until a release
is cut, as we do today). It will also help cut down on a lot of painful
manual steps today.
Reuven
On Thu, Dec 7, 2017 at 4:38 PM, Alan Myrvold <am...@google.com> wrote:
> Hi, I'm Alan.
>
> I've been working with Google Cloud engineering productivity, and I'm keen
> on improving the Beam release process and build/test infrastructure.
>
> I will be first looking into scripting some of the release validation
> steps for the nightly java snapshot releases, but hope to learn and improve
> the whole development and testing experience for Beam.
>
> Look forward to working with everyone!
>
> Alan Myrvold
>
>
Re: Introduction + interest in helping Beam builds, tests, and releases
Posted by Reuven Lax <re...@google.com>.
I think this would be very useful! Finding breaking changes early is always
better than waiting until it's too late.
Reuven
On Thu, Dec 7, 2017 at 8:00 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> Also, and I know this is maybe a bit beyond the scope of what would make
> sense initially, but if you wanted to set up something to test BEAM against
> the new Spark/Flink RCs we could give feedback about any breaking changes
> we see in upstream projects and I’d be happy to help with that :)
>
> On Fri, Dec 8, 2017 at 9:44 AM Eugene Kirpichov <ki...@google.com>
> wrote:
>
>> Awesome, excited to see release validation automated! Please let me know
>> if you need help getting Flink and Spark runner validation on Dataproc - I
>> did that manually and it involved some non-obvious steps.
>>
>> On Thu, Dec 7, 2017 at 5:29 PM Alan Myrvold <am...@google.com> wrote:
>>
>>> Hi, I'm Alan.
>>>
>>> I've been working with Google Cloud engineering productivity, and I'm
>>> keen on improving the Beam release process and build/test infrastructure.
>>>
>>> I will be first looking into scripting some of the release validation
>>> steps for the nightly java snapshot releases, but hope to learn and improve
>>> the whole development and testing experience for Beam.
>>>
>>> Look forward to working with everyone!
>>>
>>> Alan Myrvold
>>>
>>> --
> Twitter: https://twitter.com/holdenkarau
>
Re: Introduction + interest in helping Beam builds, tests, and
releases
Posted by Etienne Chauchot <ec...@apache.org>.
Welcome Alan!
Regarding the release validation automation. I'm used to launch nexmark
query set to test a release. Automating that launch would be awesome !
And maybe also comparing run times of the queries to a stored baseline
would be good too to detect performance regressions.
Etienne
Le 08/12/2017 à 15:17, Ismaël Mejía a écrit :
> Welcome Alan looking forward to your help and in particular to have
> ways to validate our releases with Hadoop's YARN too (Dataproc).
> If I can add an extra point it would be to have also some 'backwards'
> compatible version of Holden's wish.
> So we can test for example the releases with previous versions of the
> dependencies e.g. Spark 2.0, 2.1, 2.2, etc.
>
>
>
> On Fri, Dec 8, 2017 at 8:49 AM, Ahmet Altay <al...@google.com> wrote:
>> Welcome Alan, this sounds great!
>>
>> On Thu, Dec 7, 2017 at 8:00 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>>> Also, and I know this is maybe a bit beyond the scope of what would make
>>> sense initially, but if you wanted to set up something to test BEAM against
>>> the new Spark/Flink RCs we could give feedback about any breaking changes we
>>> see in upstream projects and I’d be happy to help with that :)
>>>
>>> On Fri, Dec 8, 2017 at 9:44 AM Eugene Kirpichov <ki...@google.com>
>>> wrote:
>>>> Awesome, excited to see release validation automated! Please let me know
>>>> if you need help getting Flink and Spark runner validation on Dataproc - I
>>>> did that manually and it involved some non-obvious steps.
>>>>
>>>> On Thu, Dec 7, 2017 at 5:29 PM Alan Myrvold <am...@google.com> wrote:
>>>>> Hi, I'm Alan.
>>>>>
>>>>> I've been working with Google Cloud engineering productivity, and I'm
>>>>> keen on improving the Beam release process and build/test infrastructure.
>>>>>
>>>>> I will be first looking into scripting some of the release validation
>>>>> steps for the nightly java snapshot releases, but hope to learn and improve
>>>>> the whole development and testing experience for Beam.
>>>>>
>>>>> Look forward to working with everyone!
>>>>>
>>>>> Alan Myrvold
>>>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>
Re: Introduction + interest in helping Beam builds, tests, and releases
Posted by Ismaël Mejía <ie...@gmail.com>.
Welcome Alan looking forward to your help and in particular to have
ways to validate our releases with Hadoop's YARN too (Dataproc).
If I can add an extra point it would be to have also some 'backwards'
compatible version of Holden's wish.
So we can test for example the releases with previous versions of the
dependencies e.g. Spark 2.0, 2.1, 2.2, etc.
On Fri, Dec 8, 2017 at 8:49 AM, Ahmet Altay <al...@google.com> wrote:
> Welcome Alan, this sounds great!
>
> On Thu, Dec 7, 2017 at 8:00 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>> Also, and I know this is maybe a bit beyond the scope of what would make
>> sense initially, but if you wanted to set up something to test BEAM against
>> the new Spark/Flink RCs we could give feedback about any breaking changes we
>> see in upstream projects and I’d be happy to help with that :)
>>
>> On Fri, Dec 8, 2017 at 9:44 AM Eugene Kirpichov <ki...@google.com>
>> wrote:
>>>
>>> Awesome, excited to see release validation automated! Please let me know
>>> if you need help getting Flink and Spark runner validation on Dataproc - I
>>> did that manually and it involved some non-obvious steps.
>>>
>>> On Thu, Dec 7, 2017 at 5:29 PM Alan Myrvold <am...@google.com> wrote:
>>>>
>>>> Hi, I'm Alan.
>>>>
>>>> I've been working with Google Cloud engineering productivity, and I'm
>>>> keen on improving the Beam release process and build/test infrastructure.
>>>>
>>>> I will be first looking into scripting some of the release validation
>>>> steps for the nightly java snapshot releases, but hope to learn and improve
>>>> the whole development and testing experience for Beam.
>>>>
>>>> Look forward to working with everyone!
>>>>
>>>> Alan Myrvold
>>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>
>
Re: Introduction + interest in helping Beam builds, tests, and releases
Posted by Ahmet Altay <al...@google.com>.
Welcome Alan, this sounds great!
On Thu, Dec 7, 2017 at 8:00 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> Also, and I know this is maybe a bit beyond the scope of what would make
> sense initially, but if you wanted to set up something to test BEAM against
> the new Spark/Flink RCs we could give feedback about any breaking changes
> we see in upstream projects and I’d be happy to help with that :)
>
> On Fri, Dec 8, 2017 at 9:44 AM Eugene Kirpichov <ki...@google.com>
> wrote:
>
>> Awesome, excited to see release validation automated! Please let me know
>> if you need help getting Flink and Spark runner validation on Dataproc - I
>> did that manually and it involved some non-obvious steps.
>>
>> On Thu, Dec 7, 2017 at 5:29 PM Alan Myrvold <am...@google.com> wrote:
>>
>>> Hi, I'm Alan.
>>>
>>> I've been working with Google Cloud engineering productivity, and I'm
>>> keen on improving the Beam release process and build/test infrastructure.
>>>
>>> I will be first looking into scripting some of the release validation
>>> steps for the nightly java snapshot releases, but hope to learn and improve
>>> the whole development and testing experience for Beam.
>>>
>>> Look forward to working with everyone!
>>>
>>> Alan Myrvold
>>>
>>> --
> Twitter: https://twitter.com/holdenkarau
>
Re: Introduction + interest in helping Beam builds, tests, and releases
Posted by Holden Karau <ho...@pigscanfly.ca>.
Also, and I know this is maybe a bit beyond the scope of what would make
sense initially, but if you wanted to set up something to test BEAM against
the new Spark/Flink RCs we could give feedback about any breaking changes
we see in upstream projects and I’d be happy to help with that :)
On Fri, Dec 8, 2017 at 9:44 AM Eugene Kirpichov <ki...@google.com>
wrote:
> Awesome, excited to see release validation automated! Please let me know
> if you need help getting Flink and Spark runner validation on Dataproc - I
> did that manually and it involved some non-obvious steps.
>
> On Thu, Dec 7, 2017 at 5:29 PM Alan Myrvold <am...@google.com> wrote:
>
>> Hi, I'm Alan.
>>
>> I've been working with Google Cloud engineering productivity, and I'm
>> keen on improving the Beam release process and build/test infrastructure.
>>
>> I will be first looking into scripting some of the release validation
>> steps for the nightly java snapshot releases, but hope to learn and improve
>> the whole development and testing experience for Beam.
>>
>> Look forward to working with everyone!
>>
>> Alan Myrvold
>>
>> --
Twitter: https://twitter.com/holdenkarau
Re: Introduction + interest in helping Beam builds, tests, and releases
Posted by Eugene Kirpichov <ki...@google.com>.
Awesome, excited to see release validation automated! Please let me know if
you need help getting Flink and Spark runner validation on Dataproc - I did
that manually and it involved some non-obvious steps.
On Thu, Dec 7, 2017 at 5:29 PM Alan Myrvold <am...@google.com> wrote:
> Hi, I'm Alan.
>
> I've been working with Google Cloud engineering productivity, and I'm keen
> on improving the Beam release process and build/test infrastructure.
>
> I will be first looking into scripting some of the release validation
> steps for the nightly java snapshot releases, but hope to learn and improve
> the whole development and testing experience for Beam.
>
> Look forward to working with everyone!
>
> Alan Myrvold
>
>