You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Yifan Zou <yi...@google.com> on 2019/04/04 01:34:57 UTC
Changes in Beam Jenkins Agents
Hi,
Our Jenkins are in a bad condition. 8 agents are down at this time, and
they are not going to be restored because of some bad errors happened on
the Puppet server due to the consistent re-provisioning. There are several
discussions in the recent weeks between the ASF Infra and us. In general,
the Infra team will move away from puppet third-party build nodes, and we
will need to manage the agents by ourselves.
I wrote a one pager to describe the problem and the approach. We created a
new Jenkins node (https://builds.apache.org/computer/beam17-jnlp/) for
experimental purpose. We're making efforts to verify the environment by
running all beam tests on it that make sure the required tools are
installed properly. The process is tracked in the attached spreadsheet. Any
helps on verifying tests on the new machine are appreciated! The
instructions is on the top of the spreadsheet.
One pager:
https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
Verification tracking sheet:
https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
Thanks.
Regards.
Yifan
Re: Changes in Beam Jenkins Agents
Posted by Yifan Zou <yi...@google.com>.
Thanks Kenn. Valentyn and I will verify this test.
On Fri, Apr 5, 2019 at 3:44 PM Kenneth Knowles <ke...@apache.org> wrote:
> I was just looking to grab something, and everything is marked verified or
> not a blocker except "Beam PreCommit test with Cython for Py 3.6, Py 3.7"
>
> How does one invoke this?
>
> Kenn
>
> On Thu, Apr 4, 2019 at 10:04 AM Yifan Zou <yi...@google.com> wrote:
>
>> Great! Thank you, Lukasz!
>>
>> On Thu, Apr 4, 2019 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>> I verified load tests and IO performance tests jobs. Looking
>>> good. Thanks for doing this!
>>>
>>> Łukasz
>>>
>>>
>>>
>>> czw., 4 kwi 2019 o 03:35 Yifan Zou <yi...@google.com> napisał(a):
>>>
>>>> Hi,
>>>>
>>>> Our Jenkins are in a bad condition. 8 agents are down at this time, and
>>>> they are not going to be restored because of some bad errors happened on
>>>> the Puppet server due to the consistent re-provisioning. There are several
>>>> discussions in the recent weeks between the ASF Infra and us. In general,
>>>> the Infra team will move away from puppet third-party build nodes, and we
>>>> will need to manage the agents by ourselves.
>>>>
>>>> I wrote a one pager to describe the problem and the approach. We
>>>> created a new Jenkins node (
>>>> https://builds.apache.org/computer/beam17-jnlp/) for experimental
>>>> purpose. We're making efforts to verify the environment by running all beam
>>>> tests on it that make sure the required tools are installed properly. The
>>>> process is tracked in the attached spreadsheet. Any helps on verifying
>>>> tests on the new machine are appreciated! The instructions is on the top of
>>>> the spreadsheet.
>>>>
>>>> One pager:
>>>>
>>>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
>>>>
>>>> Verification tracking sheet:
>>>>
>>>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
>>>>
>>>> Thanks.
>>>>
>>>> Regards.
>>>> Yifan
>>>>
>>>
Re: Changes in Beam Jenkins Agents
Posted by Kenneth Knowles <ke...@apache.org>.
I was just looking to grab something, and everything is marked verified or
not a blocker except "Beam PreCommit test with Cython for Py 3.6, Py 3.7"
How does one invoke this?
Kenn
On Thu, Apr 4, 2019 at 10:04 AM Yifan Zou <yi...@google.com> wrote:
> Great! Thank you, Lukasz!
>
> On Thu, Apr 4, 2019 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
>
>> I verified load tests and IO performance tests jobs. Looking good. Thanks
>> for doing this!
>>
>> Łukasz
>>
>>
>>
>> czw., 4 kwi 2019 o 03:35 Yifan Zou <yi...@google.com> napisał(a):
>>
>>> Hi,
>>>
>>> Our Jenkins are in a bad condition. 8 agents are down at this time, and
>>> they are not going to be restored because of some bad errors happened on
>>> the Puppet server due to the consistent re-provisioning. There are several
>>> discussions in the recent weeks between the ASF Infra and us. In general,
>>> the Infra team will move away from puppet third-party build nodes, and we
>>> will need to manage the agents by ourselves.
>>>
>>> I wrote a one pager to describe the problem and the approach. We created
>>> a new Jenkins node (https://builds.apache.org/computer/beam17-jnlp/)
>>> for experimental purpose. We're making efforts to verify the environment by
>>> running all beam tests on it that make sure the required tools are
>>> installed properly. The process is tracked in the attached spreadsheet. Any
>>> helps on verifying tests on the new machine are appreciated! The
>>> instructions is on the top of the spreadsheet.
>>>
>>> One pager:
>>>
>>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
>>>
>>> Verification tracking sheet:
>>>
>>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
>>>
>>> Thanks.
>>>
>>> Regards.
>>> Yifan
>>>
>>
Re: Changes in Beam Jenkins Agents
Posted by Yifan Zou <yi...@google.com>.
Great! Thank you, Lukasz!
On Thu, Apr 4, 2019 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
wrote:
> I verified load tests and IO performance tests jobs. Looking good. Thanks
> for doing this!
>
> Łukasz
>
>
>
> czw., 4 kwi 2019 o 03:35 Yifan Zou <yi...@google.com> napisał(a):
>
>> Hi,
>>
>> Our Jenkins are in a bad condition. 8 agents are down at this time, and
>> they are not going to be restored because of some bad errors happened on
>> the Puppet server due to the consistent re-provisioning. There are several
>> discussions in the recent weeks between the ASF Infra and us. In general,
>> the Infra team will move away from puppet third-party build nodes, and we
>> will need to manage the agents by ourselves.
>>
>> I wrote a one pager to describe the problem and the approach. We created
>> a new Jenkins node (https://builds.apache.org/computer/beam17-jnlp/) for
>> experimental purpose. We're making efforts to verify the environment by
>> running all beam tests on it that make sure the required tools are
>> installed properly. The process is tracked in the attached spreadsheet. Any
>> helps on verifying tests on the new machine are appreciated! The
>> instructions is on the top of the spreadsheet.
>>
>> One pager:
>>
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
>>
>> Verification tracking sheet:
>>
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
>>
>> Thanks.
>>
>> Regards.
>> Yifan
>>
>
Re: Changes in Beam Jenkins Agents
Posted by Łukasz Gajowy <lu...@gmail.com>.
I verified load tests and IO performance tests jobs. Looking good. Thanks
for doing this!
Łukasz
czw., 4 kwi 2019 o 03:35 Yifan Zou <yi...@google.com> napisał(a):
> Hi,
>
> Our Jenkins are in a bad condition. 8 agents are down at this time, and
> they are not going to be restored because of some bad errors happened on
> the Puppet server due to the consistent re-provisioning. There are several
> discussions in the recent weeks between the ASF Infra and us. In general,
> the Infra team will move away from puppet third-party build nodes, and we
> will need to manage the agents by ourselves.
>
> I wrote a one pager to describe the problem and the approach. We created a
> new Jenkins node (https://builds.apache.org/computer/beam17-jnlp/) for
> experimental purpose. We're making efforts to verify the environment by
> running all beam tests on it that make sure the required tools are
> installed properly. The process is tracked in the attached spreadsheet. Any
> helps on verifying tests on the new machine are appreciated! The
> instructions is on the top of the spreadsheet.
>
> One pager:
>
> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
>
> Verification tracking sheet:
>
> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
>
> Thanks.
>
> Regards.
> Yifan
>