You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ismaël Mejía <ie...@gmail.com> on 2019/01/16 16:17:32 UTC
Our jenkins beam1 server is down
Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
sending most builds to it so there are issues to validate some PRs.
Re: Our jenkins beam1 server is down
Posted by Rui Wang <ru...@google.com>.
Hi, seems like both Beam4[1] and Beam9[2] are down.
-Rui
[1]: https://builds.apache.org/computer/beam4/
[2]: https://builds.apache.org/computer/beam9/
On Wed, Jan 23, 2019 at 8:15 AM Yifan Zou <yi...@google.com> wrote:
> Looking. The following errors happened consistently.
>
> Jan 23 16:05:55 apache-beam-jenkins-slave-group-51fn systemd[1]: Started Session 72 of user jenkins.
> Jan 23 16:06:03 apache-beam-jenkins-slave-group-51fn snmpd[16379]: error on subcontainer 'ia_addr' insert (-1)
> Jan 23 16:08:33 apache-beam-jenkins-slave-group-51fn snmpd[16379]: message repeated 5 times: [ error on subcontainer 'ia_addr' insert (-1)]
>
>
> On Wed, Jan 23, 2019 at 7:19 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Looks like beam9 is now gone.
>>
>> On Tue, Jan 22, 2019 at 8:57 PM Yifan Zou <yi...@google.com> wrote:
>> >
>> > The inventory test on the beam1 passed. The beam1 is back to normal.
>> > https://builds.apache.org/job/beam_Inventory_beam1/303/
>> >
>> > On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou <yi...@google.com> wrote:
>> >>
>> >> Thanks for reporting the failures. Just disconnect and reconnect
>> beam1. I am creating a PR that force run a job on that agent to verify.
>> >>
>> >> On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka <go...@google.com>
>> wrote:
>> >>>
>> >>> Beam 1 seems to be down again
>> >>>
>> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
>> >>>
>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
>> >>>
>> >>> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou <yi...@google.com>
>> wrote:
>> >>>>
>> >>>> The beam1 and 14 are back and building.
>> >>>>
>> >>>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>> Thanks Yifan for taking care.
>> >>>>>
>> >>>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com>
>> wrote:
>> >>>>> >
>> >>>>> > Yes, beam14 is offline as well. We're on it.
>> >>>>> >
>> >>>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com>
>> wrote:
>> >>>>> >>
>> >>>>> >> With another try, succeeding on beam10.
>> >>>>> >>
>> >>>>> >> Thanks for the fix.
>> >>>>> >>
>> >>>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com>
>> wrote:
>> >>>>> >>>
>> >>>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
>> offline; cannot locate JDK 1.8 (latest)".
>> >>>>> >>>
>> >>>>> >>> Beam1 is not the only one broken?
>> >>>>> >>>
>> >>>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com>
>> wrote:
>> >>>>> >>>>
>> >>>>> >>>> The beam1 was still accepting jobs and breaking them after
>> reset this morning. We temporarily disconnect it so that jobs could be
>> scheduled on healthy nodes. Infra is making efforts to fix beam1.
>> >>>>> >>>>
>> >>>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <
>> yifanzou@google.com> wrote:
>> >>>>> >>>>>
>> >>>>> >>>>> The VM instance was reset and Infra is trying to repuppetize
>> it. https://issues.apache.org/jira/browse/INFRA-17672 is created to
>> track this issue.
>> >>>>> >>>>>
>> >>>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com>
>> wrote:
>> >>>>> >>>>>>
>> >>>>> >>>>>> Thanks you Yifan!
>> >>>>> >>>>>>
>> >>>>> >>>>>> Looks like following precommits are affected according to my
>> PR:
>> >>>>> >>>>>>
>> >>>>> >>>>>> Java_Examples_Dataflow,
>> >>>>> >>>>>> Portable_Python,
>> >>>>> >>>>>> Website_Stage_GCS
>> >>>>> >>>>>>
>> >>>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <
>> yifanzou@google.com> wrote:
>> >>>>> >>>>>>>
>> >>>>> >>>>>>> I am looking on it.
>> >>>>> >>>>>>>
>> >>>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <
>> iemejia@gmail.com> wrote:
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling
>> algorithm is
>> >>>>> >>>>>>>> sending most builds to it so there are issues to validate
>> some PRs.
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> --
>> >>>>> >>> ================
>> >>>>> >>> Ruoyun Huang
>> >>>>> >>>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> --
>> >>>>> >> ================
>> >>>>> >> Ruoyun Huang
>> >>>>> >>
>>
>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
Looking. The following errors happened consistently.
Jan 23 16:05:55 apache-beam-jenkins-slave-group-51fn systemd[1]:
Started Session 72 of user jenkins.
Jan 23 16:06:03 apache-beam-jenkins-slave-group-51fn snmpd[16379]:
error on subcontainer 'ia_addr' insert (-1)
Jan 23 16:08:33 apache-beam-jenkins-slave-group-51fn snmpd[16379]:
message repeated 5 times: [ error on subcontainer 'ia_addr' insert
(-1)]
On Wed, Jan 23, 2019 at 7:19 AM Ismaël Mejía <ie...@gmail.com> wrote:
> Looks like beam9 is now gone.
>
> On Tue, Jan 22, 2019 at 8:57 PM Yifan Zou <yi...@google.com> wrote:
> >
> > The inventory test on the beam1 passed. The beam1 is back to normal.
> > https://builds.apache.org/job/beam_Inventory_beam1/303/
> >
> > On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou <yi...@google.com> wrote:
> >>
> >> Thanks for reporting the failures. Just disconnect and reconnect beam1.
> I am creating a PR that force run a job on that agent to verify.
> >>
> >> On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka <go...@google.com>
> wrote:
> >>>
> >>> Beam 1 seems to be down again
> >>>
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
> >>>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
> >>>
> >>> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou <yi...@google.com>
> wrote:
> >>>>
> >>>> The beam1 and 14 are back and building.
> >>>>
> >>>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>>>
> >>>>> Thanks Yifan for taking care.
> >>>>>
> >>>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com>
> wrote:
> >>>>> >
> >>>>> > Yes, beam14 is offline as well. We're on it.
> >>>>> >
> >>>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com>
> wrote:
> >>>>> >>
> >>>>> >> With another try, succeeding on beam10.
> >>>>> >>
> >>>>> >> Thanks for the fix.
> >>>>> >>
> >>>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com>
> wrote:
> >>>>> >>>
> >>>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
> offline; cannot locate JDK 1.8 (latest)".
> >>>>> >>>
> >>>>> >>> Beam1 is not the only one broken?
> >>>>> >>>
> >>>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com>
> wrote:
> >>>>> >>>>
> >>>>> >>>> The beam1 was still accepting jobs and breaking them after
> reset this morning. We temporarily disconnect it so that jobs could be
> scheduled on healthy nodes. Infra is making efforts to fix beam1.
> >>>>> >>>>
> >>>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com>
> wrote:
> >>>>> >>>>>
> >>>>> >>>>> The VM instance was reset and Infra is trying to repuppetize
> it. https://issues.apache.org/jira/browse/INFRA-17672 is created to track
> this issue.
> >>>>> >>>>>
> >>>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com>
> wrote:
> >>>>> >>>>>>
> >>>>> >>>>>> Thanks you Yifan!
> >>>>> >>>>>>
> >>>>> >>>>>> Looks like following precommits are affected according to my
> PR:
> >>>>> >>>>>>
> >>>>> >>>>>> Java_Examples_Dataflow,
> >>>>> >>>>>> Portable_Python,
> >>>>> >>>>>> Website_Stage_GCS
> >>>>> >>>>>>
> >>>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <
> yifanzou@google.com> wrote:
> >>>>> >>>>>>>
> >>>>> >>>>>>> I am looking on it.
> >>>>> >>>>>>>
> >>>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <
> iemejia@gmail.com> wrote:
> >>>>> >>>>>>>>
> >>>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling
> algorithm is
> >>>>> >>>>>>>> sending most builds to it so there are issues to validate
> some PRs.
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> --
> >>>>> >>> ================
> >>>>> >>> Ruoyun Huang
> >>>>> >>>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> ================
> >>>>> >> Ruoyun Huang
> >>>>> >>
>
Re: Our jenkins beam1 server is down
Posted by Ismaël Mejía <ie...@gmail.com>.
Looks like beam9 is now gone.
On Tue, Jan 22, 2019 at 8:57 PM Yifan Zou <yi...@google.com> wrote:
>
> The inventory test on the beam1 passed. The beam1 is back to normal.
> https://builds.apache.org/job/beam_Inventory_beam1/303/
>
> On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou <yi...@google.com> wrote:
>>
>> Thanks for reporting the failures. Just disconnect and reconnect beam1. I am creating a PR that force run a job on that agent to verify.
>>
>> On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka <go...@google.com> wrote:
>>>
>>> Beam 1 seems to be down again
>>> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
>>>
>>> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>> The beam1 and 14 are back and building.
>>>>
>>>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>
>>>>> Thanks Yifan for taking care.
>>>>>
>>>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com> wrote:
>>>>> >
>>>>> > Yes, beam14 is offline as well. We're on it.
>>>>> >
>>>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com> wrote:
>>>>> >>
>>>>> >> With another try, succeeding on beam10.
>>>>> >>
>>>>> >> Thanks for the fix.
>>>>> >>
>>>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com> wrote:
>>>>> >>>
>>>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is offline; cannot locate JDK 1.8 (latest)".
>>>>> >>>
>>>>> >>> Beam1 is not the only one broken?
>>>>> >>>
>>>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com> wrote:
>>>>> >>>>
>>>>> >>>> The beam1 was still accepting jobs and breaking them after reset this morning. We temporarily disconnect it so that jobs could be scheduled on healthy nodes. Infra is making efforts to fix beam1.
>>>>> >>>>
>>>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com> wrote:
>>>>> >>>>>
>>>>> >>>>> The VM instance was reset and Infra is trying to repuppetize it. https://issues.apache.org/jira/browse/INFRA-17672 is created to track this issue.
>>>>> >>>>>
>>>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Thanks you Yifan!
>>>>> >>>>>>
>>>>> >>>>>> Looks like following precommits are affected according to my PR:
>>>>> >>>>>>
>>>>> >>>>>> Java_Examples_Dataflow,
>>>>> >>>>>> Portable_Python,
>>>>> >>>>>> Website_Stage_GCS
>>>>> >>>>>>
>>>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> I am looking on it.
>>>>> >>>>>>>
>>>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>>> >>>>>>>> sending most builds to it so there are issues to validate some PRs.
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> ================
>>>>> >>> Ruoyun Huang
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> ================
>>>>> >> Ruoyun Huang
>>>>> >>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
The inventory test on the beam1 passed. The beam1 is back to normal.
https://builds.apache.org/job/beam_Inventory_beam1/303/
On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou <yi...@google.com> wrote:
> Thanks for reporting the failures. Just disconnect and reconnect beam1. I
> am creating a PR that force run a job on that agent to verify.
>
> On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka <go...@google.com> wrote:
>
>> Beam 1 seems to be down again
>>
>> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
>>
>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
>>
>> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou <yi...@google.com> wrote:
>>
>>> The beam1 and 14 are back and building.
>>>
>>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Thanks Yifan for taking care.
>>>>
>>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com> wrote:
>>>> >
>>>> > Yes, beam14 is offline as well. We're on it.
>>>> >
>>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com>
>>>> wrote:
>>>> >>
>>>> >> With another try, succeeding on beam10.
>>>> >>
>>>> >> Thanks for the fix.
>>>> >>
>>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com>
>>>> wrote:
>>>> >>>
>>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
>>>> offline; cannot locate JDK 1.8 (latest)".
>>>> >>>
>>>> >>> Beam1 is not the only one broken?
>>>> >>>
>>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> The beam1 was still accepting jobs and breaking them after reset
>>>> this morning. We temporarily disconnect it so that jobs could be scheduled
>>>> on healthy nodes. Infra is making efforts to fix beam1.
>>>> >>>>
>>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
>>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>>> this issue.
>>>> >>>>>
>>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com>
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Thanks you Yifan!
>>>> >>>>>>
>>>> >>>>>> Looks like following precommits are affected according to my PR:
>>>> >>>>>>
>>>> >>>>>> Java_Examples_Dataflow,
>>>> >>>>>> Portable_Python,
>>>> >>>>>> Website_Stage_GCS
>>>> >>>>>>
>>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com>
>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> I am looking on it.
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm
>>>> is
>>>> >>>>>>>> sending most builds to it so there are issues to validate some
>>>> PRs.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> ================
>>>> >>> Ruoyun Huang
>>>> >>>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> ================
>>>> >> Ruoyun Huang
>>>> >>
>>>>
>>>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
Thanks for reporting the failures. Just disconnect and reconnect beam1. I
am creating a PR that force run a job on that agent to verify.
On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka <go...@google.com> wrote:
> Beam 1 seems to be down again
>
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
>
> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou <yi...@google.com> wrote:
>
>> The beam1 and 14 are back and building.
>>
>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Thanks Yifan for taking care.
>>>
>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com> wrote:
>>> >
>>> > Yes, beam14 is offline as well. We're on it.
>>> >
>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com>
>>> wrote:
>>> >>
>>> >> With another try, succeeding on beam10.
>>> >>
>>> >> Thanks for the fix.
>>> >>
>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com>
>>> wrote:
>>> >>>
>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
>>> offline; cannot locate JDK 1.8 (latest)".
>>> >>>
>>> >>> Beam1 is not the only one broken?
>>> >>>
>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> The beam1 was still accepting jobs and breaking them after reset
>>> this morning. We temporarily disconnect it so that jobs could be scheduled
>>> on healthy nodes. Infra is making efforts to fix beam1.
>>> >>>>
>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com>
>>> wrote:
>>> >>>>>
>>> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>> this issue.
>>> >>>>>
>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> Thanks you Yifan!
>>> >>>>>>
>>> >>>>>> Looks like following precommits are affected according to my PR:
>>> >>>>>>
>>> >>>>>> Java_Examples_Dataflow,
>>> >>>>>> Portable_Python,
>>> >>>>>> Website_Stage_GCS
>>> >>>>>>
>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com>
>>> wrote:
>>> >>>>>>>
>>> >>>>>>> I am looking on it.
>>> >>>>>>>
>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com>
>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>> >>>>>>>> sending most builds to it so there are issues to validate some
>>> PRs.
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> ================
>>> >>> Ruoyun Huang
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> ================
>>> >> Ruoyun Huang
>>> >>
>>>
>>
Re: Our jenkins beam1 server is down
Posted by Ankur Goenka <go...@google.com>.
Beam 1 seems to be down again
https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou <yi...@google.com> wrote:
> The beam1 and 14 are back and building.
>
> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Thanks Yifan for taking care.
>>
>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com> wrote:
>> >
>> > Yes, beam14 is offline as well. We're on it.
>> >
>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com> wrote:
>> >>
>> >> With another try, succeeding on beam10.
>> >>
>> >> Thanks for the fix.
>> >>
>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com>
>> wrote:
>> >>>
>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
>> offline; cannot locate JDK 1.8 (latest)".
>> >>>
>> >>> Beam1 is not the only one broken?
>> >>>
>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com>
>> wrote:
>> >>>>
>> >>>> The beam1 was still accepting jobs and breaking them after reset
>> this morning. We temporarily disconnect it so that jobs could be scheduled
>> on healthy nodes. Infra is making efforts to fix beam1.
>> >>>>
>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com>
>> wrote:
>> >>>>>
>> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>> this issue.
>> >>>>>
>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com>
>> wrote:
>> >>>>>>
>> >>>>>> Thanks you Yifan!
>> >>>>>>
>> >>>>>> Looks like following precommits are affected according to my PR:
>> >>>>>>
>> >>>>>> Java_Examples_Dataflow,
>> >>>>>> Portable_Python,
>> >>>>>> Website_Stage_GCS
>> >>>>>>
>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com>
>> wrote:
>> >>>>>>>
>> >>>>>>> I am looking on it.
>> >>>>>>>
>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>>>>>>
>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>> >>>>>>>> sending most builds to it so there are issues to validate some
>> PRs.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> ================
>> >>> Ruoyun Huang
>> >>>
>> >>
>> >>
>> >> --
>> >> ================
>> >> Ruoyun Huang
>> >>
>>
>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
The beam1 and 14 are back and building.
On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía <ie...@gmail.com> wrote:
> Thanks Yifan for taking care.
>
> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com> wrote:
> >
> > Yes, beam14 is offline as well. We're on it.
> >
> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com> wrote:
> >>
> >> With another try, succeeding on beam10.
> >>
> >> Thanks for the fix.
> >>
> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com> wrote:
> >>>
> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is offline;
> cannot locate JDK 1.8 (latest)".
> >>>
> >>> Beam1 is not the only one broken?
> >>>
> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com> wrote:
> >>>>
> >>>> The beam1 was still accepting jobs and breaking them after reset this
> morning. We temporarily disconnect it so that jobs could be scheduled on
> healthy nodes. Infra is making efforts to fix beam1.
> >>>>
> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com>
> wrote:
> >>>>>
> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
> this issue.
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com>
> wrote:
> >>>>>>
> >>>>>> Thanks you Yifan!
> >>>>>>
> >>>>>> Looks like following precommits are affected according to my PR:
> >>>>>>
> >>>>>> Java_Examples_Dataflow,
> >>>>>> Portable_Python,
> >>>>>> Website_Stage_GCS
> >>>>>>
> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com>
> wrote:
> >>>>>>>
> >>>>>>> I am looking on it.
> >>>>>>>
> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
> >>>>>>>> sending most builds to it so there are issues to validate some
> PRs.
> >>>
> >>>
> >>>
> >>> --
> >>> ================
> >>> Ruoyun Huang
> >>>
> >>
> >>
> >> --
> >> ================
> >> Ruoyun Huang
> >>
>
Re: Our jenkins beam1 server is down
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks Yifan for taking care.
On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou <yi...@google.com> wrote:
>
> Yes, beam14 is offline as well. We're on it.
>
> On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com> wrote:
>>
>> With another try, succeeding on beam10.
>>
>> Thanks for the fix.
>>
>> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com> wrote:
>>>
>>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is offline; cannot locate JDK 1.8 (latest)".
>>>
>>> Beam1 is not the only one broken?
>>>
>>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>> The beam1 was still accepting jobs and breaking them after reset this morning. We temporarily disconnect it so that jobs could be scheduled on healthy nodes. Infra is making efforts to fix beam1.
>>>>
>>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com> wrote:
>>>>>
>>>>> The VM instance was reset and Infra is trying to repuppetize it. https://issues.apache.org/jira/browse/INFRA-17672 is created to track this issue.
>>>>>
>>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
>>>>>>
>>>>>> Thanks you Yifan!
>>>>>>
>>>>>> Looks like following precommits are affected according to my PR:
>>>>>>
>>>>>> Java_Examples_Dataflow,
>>>>>> Portable_Python,
>>>>>> Website_Stage_GCS
>>>>>>
>>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>>>>>>>
>>>>>>> I am looking on it.
>>>>>>>
>>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>>>>>> sending most builds to it so there are issues to validate some PRs.
>>>
>>>
>>>
>>> --
>>> ================
>>> Ruoyun Huang
>>>
>>
>>
>> --
>> ================
>> Ruoyun Huang
>>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
Yes, beam14 is offline as well. We're on it.
On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang <ru...@google.com> wrote:
> With another try, succeeding on beam10.
>
> Thanks for the fix.
>
> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com> wrote:
>
>> Just did a rerun, got error saying "*10:12:21* ERROR: beam14 is offline;
>> cannot locate JDK 1.8 (latest)".
>>
>> Beam1 is not the only one broken?
>>
>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com> wrote:
>>
>>> The beam1 was still accepting jobs and breaking them after reset this
>>> morning. We temporarily disconnect it so that jobs could be scheduled on
>>> healthy nodes. Infra is making efforts to fix beam1.
>>>
>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com> wrote:
>>>
>>>> The VM instance was reset and Infra is trying to repuppetize it.
>>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>>> this issue.
>>>>
>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
>>>>
>>>>> Thanks you Yifan!
>>>>>
>>>>> Looks like following precommits are affected according to my PR:
>>>>>
>>>>> Java_Examples_Dataflow,
>>>>> Portable_Python,
>>>>> Website_Stage_GCS
>>>>>
>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>>>>>
>>>>>> I am looking on it.
>>>>>>
>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>>>>> sending most builds to it so there are issues to validate some PRs.
>>>>>>>
>>>>>>
>>
>> --
>> ================
>> Ruoyun Huang
>>
>>
>
> --
> ================
> Ruoyun Huang
>
>
Re: Our jenkins beam1 server is down
Posted by Ruoyun Huang <ru...@google.com>.
With another try, succeeding on beam10.
Thanks for the fix.
On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang <ru...@google.com> wrote:
> Just did a rerun, got error saying "*10:12:21* ERROR: beam14 is offline;
> cannot locate JDK 1.8 (latest)".
>
> Beam1 is not the only one broken?
>
> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com> wrote:
>
>> The beam1 was still accepting jobs and breaking them after reset this
>> morning. We temporarily disconnect it so that jobs could be scheduled on
>> healthy nodes. Infra is making efforts to fix beam1.
>>
>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com> wrote:
>>
>>> The VM instance was reset and Infra is trying to repuppetize it.
>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>> this issue.
>>>
>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
>>>
>>>> Thanks you Yifan!
>>>>
>>>> Looks like following precommits are affected according to my PR:
>>>>
>>>> Java_Examples_Dataflow,
>>>> Portable_Python,
>>>> Website_Stage_GCS
>>>>
>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>>> I am looking on it.
>>>>>
>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>>>> sending most builds to it so there are issues to validate some PRs.
>>>>>>
>>>>>
>
> --
> ================
> Ruoyun Huang
>
>
--
================
Ruoyun Huang
Re: Our jenkins beam1 server is down
Posted by Ruoyun Huang <ru...@google.com>.
Just did a rerun, got error saying "*10:12:21* ERROR: beam14 is offline;
cannot locate JDK 1.8 (latest)".
Beam1 is not the only one broken?
On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou <yi...@google.com> wrote:
> The beam1 was still accepting jobs and breaking them after reset this
> morning. We temporarily disconnect it so that jobs could be scheduled on
> healthy nodes. Infra is making efforts to fix beam1.
>
> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com> wrote:
>
>> The VM instance was reset and Infra is trying to repuppetize it.
>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>> this issue.
>>
>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
>>
>>> Thanks you Yifan!
>>>
>>> Looks like following precommits are affected according to my PR:
>>>
>>> Java_Examples_Dataflow,
>>> Portable_Python,
>>> Website_Stage_GCS
>>>
>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>>>
>>>> I am looking on it.
>>>>
>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>>> sending most builds to it so there are issues to validate some PRs.
>>>>>
>>>>
--
================
Ruoyun Huang
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
The beam1 was still accepting jobs and breaking them after reset this
morning. We temporarily disconnect it so that jobs could be scheduled on
healthy nodes. Infra is making efforts to fix beam1.
On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou <yi...@google.com> wrote:
> The VM instance was reset and Infra is trying to repuppetize it.
> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
> this issue.
>
> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
>
>> Thanks you Yifan!
>>
>> Looks like following precommits are affected according to my PR:
>>
>> Java_Examples_Dataflow,
>> Portable_Python,
>> Website_Stage_GCS
>>
>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>>
>>> I am looking on it.
>>>
>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>> sending most builds to it so there are issues to validate some PRs.
>>>>
>>>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
The VM instance was reset and Infra is trying to repuppetize it.
https://issues.apache.org/jira/browse/INFRA-17672 is created to track this
issue.
On Wed, Jan 16, 2019 at 10:51 AM Mark Liu <ma...@google.com> wrote:
> Thanks you Yifan!
>
> Looks like following precommits are affected according to my PR:
>
> Java_Examples_Dataflow,
> Portable_Python,
> Website_Stage_GCS
>
> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
>
>> I am looking on it.
>>
>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>> sending most builds to it so there are issues to validate some PRs.
>>>
>>
Re: Our jenkins beam1 server is down
Posted by Mark Liu <ma...@google.com>.
Thanks you Yifan!
Looks like following precommits are affected according to my PR:
Java_Examples_Dataflow,
Portable_Python,
Website_Stage_GCS
On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <yi...@google.com> wrote:
> I am looking on it.
>
> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>> sending most builds to it so there are issues to validate some PRs.
>>
>
Re: Our jenkins beam1 server is down
Posted by Yifan Zou <yi...@google.com>.
I am looking on it.
On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
> sending most builds to it so there are issues to validate some PRs.
>