You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Daniela Martín <da...@wizeline.com> on 2022/01/26 18:30:37 UTC
Upgrading Jenkins Workers BEAM-12621
Hi everyone,
We are updating the Jenkins workers with a new image that was upgraded from
Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more details on:
BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>
If something comes up with the workers or the tests please reach out to us.
Please let us know if you have any questions or comments.
Thank you.
Regards,
--
Daniela Martín (She/Her) | <https://www.wizeline.com/>
Site Reliability Engineer III
daniela.martin@wizeline.com
Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
<https://www.facebook.com/WizelineGlobal> | Instagram
<https://www.instagram.com/wizelineglobal/> | LinkedIn
<https://www.linkedin.com/company/wizeline>
Share feedback on Clutch <https://clutch.co/review/submit/375119>
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Kiley Sok <ki...@google.com>.
Here's the doc with the details of the Jenkins update. Please leave
comments for future updates!
https://docs.google.com/document/d/19AllIonqtBBx5mXrNaUQCe3tAul7GCeT6HBrNpwE1kk/edit#
On Thu, Feb 3, 2022 at 1:49 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> also, i'd do a google doc instead of pdf so that the content is
> commentable by the community
>
> On Thu, Feb 3, 2022 at 1:46 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Took a quick look. I am not sure the below part is accurate:
>>
>> According to the Python
>> Tips Wiki, we have the python#.#-distutils dependency that it can only be
>> installed at
>> an Operating System level because it’s a deprecated package and its only
>> way to be
>> installed is through the system.
>> Pyenv can’t access those packages by its self-contained installation
>> nature and it is not
>> possible to install distutils dependents packages, such as python-apt.
>> Also, the
>> following packages are not supported/found anymore by pyenv pip installer:
>>
>>
>> I left my thoughts on that in previous emails.
>>
>> On Tue, Feb 1, 2022 at 2:43 PM Daniela Martín <
>> daniela.martin@wizeline.com> wrote:
>>
>>> Thank you everyone for all the help and patience. We will be on the
>>> lookout if something comes up with the Jenkins jobs.
>>>
>>> In the next few days, we will share with you more details of the work
>>> done in the upgrade.
>>>
>>> Thank you.
>>>
>>> Regards,
>>>
>>> On Tue, Feb 1, 2022 at 3:08 PM Kiley Sok <ki...@google.com> wrote:
>>>
>>>> The rest of the workers were updated to the new image and are back
>>>> online. Thanks for your patience!
>>>>
>>>> On Mon, Jan 31, 2022 at 4:43 PM Daniela Martín <
>>>> daniela.martin@wizeline.com> wrote:
>>>>
>>>>> Thanks Kiley, it seems that the failures are not related to the *Ubuntu
>>>>> 20.04.3* upgrade.
>>>>>
>>>>> In case the *Python Release Candidate* job works, we can proceed with
>>>>> the upgrade of the remaining Jenkins workers.
>>>>>
>>>>> Thank you.
>>>>>
>>>>> Regards,
>>>>>
>>>>> On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <ki...@google.com> wrote:
>>>>>
>>>>>> XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
>>>>>>
>>>>>> On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <
>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> Thank you for the information and details.
>>>>>>>
>>>>>>> We reviewed the tests and we are taking a look at the following jobs
>>>>>>> that started failing close to the date the Jenkins instance was upgraded.
>>>>>>> We are not sure if it’s because of the Ubuntu 20.04.3 upgrade or for
>>>>>>> another reason.
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> beam_PostCommit_XVR_Direct
>>>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
>>>>>>> -
>>>>>>>
>>>>>>> beam_PostCommit_XVR_Flink
>>>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
>>>>>>> -
>>>>>>>
>>>>>>> beam_PostCommit_XVR_Spark3
>>>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>>>>>>>
>>>>>>>
>>>>>>> We will continue testing them. Any information regarding these jobs
>>>>>>> would be greatly appreciated.
>>>>>>>
>>>>>>> Emily, could you please re-run the Python Release Candidate
>>>>>>> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/>
>>>>>>> job in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>>>>>>>
>>>>>>>
>>>>>>> Thank you very much.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Hey Daniela and Giomar (& all), do you see any concerns with the
>>>>>>>> current image so far? If not, let's upgrade the remaining workers 9-15 to
>>>>>>>> the new image. Thanks!
>>>>>>>>
>>>>>>>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll
>>>>>>>>> keep the pool at 50% capacity with the benched workers still on ubuntu 16
>>>>>>>>> as backup and either upgrade all of the workers to ubuntu 20 or roll back
>>>>>>>>> again to ubuntu 16 next week.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Nexmark failures are from an unrelated change, fyi
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> It looks like something is resetting the *Restrict where this
>>>>>>>>>>> project can be run* label setting on
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>>>>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>>>>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>>>>>>>
>>>>>>>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>>>>>>>> broken by the update:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>>>>>>>
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>>>>>>>
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>>>>>>>
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I found them by looking at https://ci-beam.apache.org/ and
>>>>>>>>>>> sorting by status. Looks like we also have a lot of permared Jenkins
>>>>>>>>>>> suites...
>>>>>>>>>>>
>>>>>>>>>>> Also, we should maybe think of some sustainable rolling-upgrades
>>>>>>>>>>> for Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>>>>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'm also running into these errors on my PR for testing the
>>>>>>>>>>>> release https://github.com/apache/beam/pull/16632, re-running
>>>>>>>>>>>> doesn't seem to help. Do we have to force the machine per-job/task?
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thinking some more, the correct course of action may also be
>>>>>>>>>>>>> to fix the job to install necessary Python packages if they are missing.
>>>>>>>>>>>>> This would make it self-contained, and the job would install them in the
>>>>>>>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I
>>>>>>>>>>>>>> expected worker #14 that runs ubuntu16. I reconfigured the labels, not sure
>>>>>>>>>>>>>> why my changes were not saved initially, and restarted the test on your PR:
>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <
>>>>>>>>>>>>>> bhulette@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure
>>>>>>>>>>>>>>> but it failed again with the same error:
>>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We are updating the Jenkins workers with a new image that
>>>>>>>>>>>>>>>>> was upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If something comes up with the workers or the tests please
>>>>>>>>>>>>>>>>> reach out to us.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050
>>>>>>>>>>>>>>>>> Zapopan, Jal.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> |
>>>>>>>>>>>>>>>>> Instagram <https://www.instagram.com/wizelineglobal/> |
>>>>>>>>>>>>>>>>> LinkedIn <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> *This email and its contents (including any attachments)
>>>>>>>>>>>>>>>>> are being sent toyou on the condition of confidentiality and may be
>>>>>>>>>>>>>>>>> protected by legalprivilege. Access to this email by anyone other than the
>>>>>>>>>>>>>>>>> intended recipientis unauthorized. If you are not the intended recipient,
>>>>>>>>>>>>>>>>> please immediatelynotify the sender by replying to this message and delete
>>>>>>>>>>>>>>>>> the materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>>>>>>>> email.*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>
>>>>>>> Site Reliability Engineer III
>>>>>>>
>>>>>>> daniela.martin@wizeline.com
>>>>>>>
>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>>>
>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>
>>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *This email and its contents (including any attachments) are being
>>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>> email.*
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>
>>>>> Site Reliability Engineer III
>>>>>
>>>>> daniela.martin@wizeline.com
>>>>>
>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>
>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>
>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *This email and its contents (including any attachments) are being
>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>> norepresentation is made with respect to any content contained in this
>>>>> email.*
>>>>
>>>>
>>>
>>> --
>>>
>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>
>>> Site Reliability Engineer III
>>>
>>> daniela.martin@wizeline.com
>>>
>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>
>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>> <https://www.linkedin.com/company/wizeline>
>>>
>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
also, i'd do a google doc instead of pdf so that the content is
commentable by the community
On Thu, Feb 3, 2022 at 1:46 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Took a quick look. I am not sure the below part is accurate:
>
> According to the Python
> Tips Wiki, we have the python#.#-distutils dependency that it can only be
> installed at
> an Operating System level because it’s a deprecated package and its only
> way to be
> installed is through the system.
> Pyenv can’t access those packages by its self-contained installation
> nature and it is not
> possible to install distutils dependents packages, such as python-apt.
> Also, the
> following packages are not supported/found anymore by pyenv pip installer:
>
>
> I left my thoughts on that in previous emails.
>
> On Tue, Feb 1, 2022 at 2:43 PM Daniela Martín <da...@wizeline.com>
> wrote:
>
>> Thank you everyone for all the help and patience. We will be on the
>> lookout if something comes up with the Jenkins jobs.
>>
>> In the next few days, we will share with you more details of the work
>> done in the upgrade.
>>
>> Thank you.
>>
>> Regards,
>>
>> On Tue, Feb 1, 2022 at 3:08 PM Kiley Sok <ki...@google.com> wrote:
>>
>>> The rest of the workers were updated to the new image and are back
>>> online. Thanks for your patience!
>>>
>>> On Mon, Jan 31, 2022 at 4:43 PM Daniela Martín <
>>> daniela.martin@wizeline.com> wrote:
>>>
>>>> Thanks Kiley, it seems that the failures are not related to the *Ubuntu
>>>> 20.04.3* upgrade.
>>>>
>>>> In case the *Python Release Candidate* job works, we can proceed with
>>>> the upgrade of the remaining Jenkins workers.
>>>>
>>>> Thank you.
>>>>
>>>> Regards,
>>>>
>>>> On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <ki...@google.com> wrote:
>>>>
>>>>> XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
>>>>>
>>>>> On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <
>>>>> daniela.martin@wizeline.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Thank you for the information and details.
>>>>>>
>>>>>> We reviewed the tests and we are taking a look at the following jobs
>>>>>> that started failing close to the date the Jenkins instance was upgraded.
>>>>>> We are not sure if it’s because of the Ubuntu 20.04.3 upgrade or for
>>>>>> another reason.
>>>>>>
>>>>>>
>>>>>> -
>>>>>>
>>>>>> beam_PostCommit_XVR_Direct
>>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
>>>>>> -
>>>>>>
>>>>>> beam_PostCommit_XVR_Flink
>>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
>>>>>> -
>>>>>>
>>>>>> beam_PostCommit_XVR_Spark3
>>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>>>>>>
>>>>>>
>>>>>> We will continue testing them. Any information regarding these jobs
>>>>>> would be greatly appreciated.
>>>>>>
>>>>>> Emily, could you please re-run the Python Release Candidate
>>>>>> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/>
>>>>>> job in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>>>>>>
>>>>>>
>>>>>> Thank you very much.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Hey Daniela and Giomar (& all), do you see any concerns with the
>>>>>>> current image so far? If not, let's upgrade the remaining workers 9-15 to
>>>>>>> the new image. Thanks!
>>>>>>>
>>>>>>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll
>>>>>>>> keep the pool at 50% capacity with the benched workers still on ubuntu 16
>>>>>>>> as backup and either upgrade all of the workers to ubuntu 20 or roll back
>>>>>>>> again to ubuntu 16 next week.
>>>>>>>>
>>>>>>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Nexmark failures are from an unrelated change, fyi
>>>>>>>>>
>>>>>>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> It looks like something is resetting the *Restrict where this
>>>>>>>>>> project can be run* label setting on
>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>>>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>>>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>>>>>>
>>>>>>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>>>>>>> broken by the update:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>>>>>>
>>>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>>>>>>
>>>>>>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>>>>>>
>>>>>>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I found them by looking at https://ci-beam.apache.org/ and
>>>>>>>>>> sorting by status. Looks like we also have a lot of permared Jenkins
>>>>>>>>>> suites...
>>>>>>>>>>
>>>>>>>>>> Also, we should maybe think of some sustainable rolling-upgrades
>>>>>>>>>> for Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>>>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm also running into these errors on my PR for testing the
>>>>>>>>>>> release https://github.com/apache/beam/pull/16632, re-running
>>>>>>>>>>> doesn't seem to help. Do we have to force the machine per-job/task?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thinking some more, the correct course of action may also be to
>>>>>>>>>>>> fix the job to install necessary Python packages if they are missing. This
>>>>>>>>>>>> would make it self-contained, and the job would install them in the
>>>>>>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <
>>>>>>>>>>>>> bhulette@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but
>>>>>>>>>>>>>> it failed again with the same error:
>>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We are updating the Jenkins workers with a new image that
>>>>>>>>>>>>>>>> was upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If something comes up with the workers or the tests please
>>>>>>>>>>>>>>>> reach out to us.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050
>>>>>>>>>>>>>>>> Zapopan, Jal.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> |
>>>>>>>>>>>>>>>> Instagram <https://www.instagram.com/wizelineglobal/> |
>>>>>>>>>>>>>>>> LinkedIn <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> *This email and its contents (including any attachments)
>>>>>>>>>>>>>>>> are being sent toyou on the condition of confidentiality and may be
>>>>>>>>>>>>>>>> protected by legalprivilege. Access to this email by anyone other than the
>>>>>>>>>>>>>>>> intended recipientis unauthorized. If you are not the intended recipient,
>>>>>>>>>>>>>>>> please immediatelynotify the sender by replying to this message and delete
>>>>>>>>>>>>>>>> the materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>>>>>>> email.*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>
>>>>>> Site Reliability Engineer III
>>>>>>
>>>>>> daniela.martin@wizeline.com
>>>>>>
>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>>
>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>
>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *This email and its contents (including any attachments) are being
>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>> norepresentation is made with respect to any content contained in this
>>>>>> email.*
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>
>>>> Site Reliability Engineer III
>>>>
>>>> daniela.martin@wizeline.com
>>>>
>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>
>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>> <https://www.linkedin.com/company/wizeline>
>>>>
>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *This email and its contents (including any attachments) are being sent
>>>> toyou on the condition of confidentiality and may be protected by
>>>> legalprivilege. Access to this email by anyone other than the intended
>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>> immediatelynotify the sender by replying to this message and delete the
>>>> materialimmediately from your system. Any further use, dissemination,
>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>> norepresentation is made with respect to any content contained in this
>>>> email.*
>>>
>>>
>>
>> --
>>
>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>
>> Site Reliability Engineer III
>>
>> daniela.martin@wizeline.com
>>
>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>
>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>> <https://www.facebook.com/WizelineGlobal> | Instagram
>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>> <https://www.linkedin.com/company/wizeline>
>>
>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
Took a quick look. I am not sure the below part is accurate:
According to the Python
Tips Wiki, we have the python#.#-distutils dependency that it can only be
installed at
an Operating System level because it’s a deprecated package and its only
way to be
installed is through the system.
Pyenv can’t access those packages by its self-contained installation nature
and it is not
possible to install distutils dependents packages, such as python-apt.
Also, the
following packages are not supported/found anymore by pyenv pip installer:
I left my thoughts on that in previous emails.
On Tue, Feb 1, 2022 at 2:43 PM Daniela Martín <da...@wizeline.com>
wrote:
> Thank you everyone for all the help and patience. We will be on the
> lookout if something comes up with the Jenkins jobs.
>
> In the next few days, we will share with you more details of the work done
> in the upgrade.
>
> Thank you.
>
> Regards,
>
> On Tue, Feb 1, 2022 at 3:08 PM Kiley Sok <ki...@google.com> wrote:
>
>> The rest of the workers were updated to the new image and are back
>> online. Thanks for your patience!
>>
>> On Mon, Jan 31, 2022 at 4:43 PM Daniela Martín <
>> daniela.martin@wizeline.com> wrote:
>>
>>> Thanks Kiley, it seems that the failures are not related to the *Ubuntu
>>> 20.04.3* upgrade.
>>>
>>> In case the *Python Release Candidate* job works, we can proceed with
>>> the upgrade of the remaining Jenkins workers.
>>>
>>> Thank you.
>>>
>>> Regards,
>>>
>>> On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <ki...@google.com> wrote:
>>>
>>>> XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
>>>>
>>>> On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <
>>>> daniela.martin@wizeline.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Thank you for the information and details.
>>>>>
>>>>> We reviewed the tests and we are taking a look at the following jobs
>>>>> that started failing close to the date the Jenkins instance was upgraded.
>>>>> We are not sure if it’s because of the Ubuntu 20.04.3 upgrade or for
>>>>> another reason.
>>>>>
>>>>>
>>>>> -
>>>>>
>>>>> beam_PostCommit_XVR_Direct
>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
>>>>> -
>>>>>
>>>>> beam_PostCommit_XVR_Flink
>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
>>>>> -
>>>>>
>>>>> beam_PostCommit_XVR_Spark3
>>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>>>>>
>>>>>
>>>>> We will continue testing them. Any information regarding these jobs
>>>>> would be greatly appreciated.
>>>>>
>>>>> Emily, could you please re-run the Python Release Candidate
>>>>> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/>
>>>>> job in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>>>>>
>>>>>
>>>>> Thank you very much.
>>>>>
>>>>> Regards,
>>>>>
>>>>> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Hey Daniela and Giomar (& all), do you see any concerns with the
>>>>>> current image so far? If not, let's upgrade the remaining workers 9-15 to
>>>>>> the new image. Thanks!
>>>>>>
>>>>>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep
>>>>>>> the pool at 50% capacity with the benched workers still on ubuntu 16 as
>>>>>>> backup and either upgrade all of the workers to ubuntu 20 or roll back
>>>>>>> again to ubuntu 16 next week.
>>>>>>>
>>>>>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Nexmark failures are from an unrelated change, fyi
>>>>>>>>
>>>>>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> It looks like something is resetting the *Restrict where this
>>>>>>>>> project can be run* label setting on
>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>>>>>
>>>>>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>>>>>> broken by the update:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>>>>>
>>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>>>>>
>>>>>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>>>>>
>>>>>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>>>>>
>>>>>>>>> I found them by looking at https://ci-beam.apache.org/ and
>>>>>>>>> sorting by status. Looks like we also have a lot of permared Jenkins
>>>>>>>>> suites...
>>>>>>>>>
>>>>>>>>> Also, we should maybe think of some sustainable rolling-upgrades
>>>>>>>>> for Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I'm also running into these errors on my PR for testing the
>>>>>>>>>> release https://github.com/apache/beam/pull/16632, re-running
>>>>>>>>>> doesn't seem to help. Do we have to force the machine per-job/task?
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thinking some more, the correct course of action may also be to
>>>>>>>>>>> fix the job to install necessary Python packages if they are missing. This
>>>>>>>>>>> would make it self-contained, and the job would install them in the
>>>>>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <
>>>>>>>>>>>> bhulette@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but
>>>>>>>>>>>>> it failed again with the same error:
>>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We are updating the Jenkins workers with a new image that
>>>>>>>>>>>>>>> was upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If something comes up with the workers or the tests please
>>>>>>>>>>>>>>> reach out to us.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050
>>>>>>>>>>>>>>> Zapopan, Jal.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> |
>>>>>>>>>>>>>>> Instagram <https://www.instagram.com/wizelineglobal/> |
>>>>>>>>>>>>>>> LinkedIn <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *This email and its contents (including any attachments) are
>>>>>>>>>>>>>>> being sent toyou on the condition of confidentiality and may be protected
>>>>>>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>>>>>> email.*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>
>>>>> Site Reliability Engineer III
>>>>>
>>>>> daniela.martin@wizeline.com
>>>>>
>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>
>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>
>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *This email and its contents (including any attachments) are being
>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>> norepresentation is made with respect to any content contained in this
>>>>> email.*
>>>>
>>>>
>>>
>>> --
>>>
>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>
>>> Site Reliability Engineer III
>>>
>>> daniela.martin@wizeline.com
>>>
>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>
>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>> <https://www.linkedin.com/company/wizeline>
>>>
>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>
>
> --
>
> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>
> Site Reliability Engineer III
>
> daniela.martin@wizeline.com
>
> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>
> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
> <https://www.facebook.com/WizelineGlobal> | Instagram
> <https://www.instagram.com/wizelineglobal/> | LinkedIn
> <https://www.linkedin.com/company/wizeline>
>
> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Daniela Martín <da...@wizeline.com>.
Thank you everyone for all the help and patience. We will be on the lookout
if something comes up with the Jenkins jobs.
In the next few days, we will share with you more details of the work done
in the upgrade.
Thank you.
Regards,
On Tue, Feb 1, 2022 at 3:08 PM Kiley Sok <ki...@google.com> wrote:
> The rest of the workers were updated to the new image and are back online.
> Thanks for your patience!
>
> On Mon, Jan 31, 2022 at 4:43 PM Daniela Martín <
> daniela.martin@wizeline.com> wrote:
>
>> Thanks Kiley, it seems that the failures are not related to the *Ubuntu
>> 20.04.3* upgrade.
>>
>> In case the *Python Release Candidate* job works, we can proceed with
>> the upgrade of the remaining Jenkins workers.
>>
>> Thank you.
>>
>> Regards,
>>
>> On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <ki...@google.com> wrote:
>>
>>> XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
>>>
>>> On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <
>>> daniela.martin@wizeline.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Thank you for the information and details.
>>>>
>>>> We reviewed the tests and we are taking a look at the following jobs
>>>> that started failing close to the date the Jenkins instance was upgraded.
>>>> We are not sure if it’s because of the Ubuntu 20.04.3 upgrade or for
>>>> another reason.
>>>>
>>>>
>>>> -
>>>>
>>>> beam_PostCommit_XVR_Direct
>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
>>>> -
>>>>
>>>> beam_PostCommit_XVR_Flink
>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
>>>> -
>>>>
>>>> beam_PostCommit_XVR_Spark3
>>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>>>>
>>>>
>>>> We will continue testing them. Any information regarding these jobs
>>>> would be greatly appreciated.
>>>>
>>>> Emily, could you please re-run the Python Release Candidate
>>>> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/>
>>>> job in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>>>>
>>>>
>>>> Thank you very much.
>>>>
>>>> Regards,
>>>>
>>>> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Hey Daniela and Giomar (& all), do you see any concerns with the
>>>>> current image so far? If not, let's upgrade the remaining workers 9-15 to
>>>>> the new image. Thanks!
>>>>>
>>>>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep
>>>>>> the pool at 50% capacity with the benched workers still on ubuntu 16 as
>>>>>> backup and either upgrade all of the workers to ubuntu 20 or roll back
>>>>>> again to ubuntu 16 next week.
>>>>>>
>>>>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Nexmark failures are from an unrelated change, fyi
>>>>>>>
>>>>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> It looks like something is resetting the *Restrict where this
>>>>>>>> project can be run* label setting on
>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>>>>
>>>>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>>>>> broken by the update:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>>>>
>>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>>>>
>>>>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>>>>
>>>>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>>>>
>>>>>>>> I found them by looking at https://ci-beam.apache.org/ and
>>>>>>>> sorting by status. Looks like we also have a lot of permared Jenkins
>>>>>>>> suites...
>>>>>>>>
>>>>>>>> Also, we should maybe think of some sustainable rolling-upgrades
>>>>>>>> for Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm also running into these errors on my PR for testing the
>>>>>>>>> release https://github.com/apache/beam/pull/16632, re-running
>>>>>>>>> doesn't seem to help. Do we have to force the machine per-job/task?
>>>>>>>>>
>>>>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thinking some more, the correct course of action may also be to
>>>>>>>>>> fix the job to install necessary Python packages if they are missing. This
>>>>>>>>>> would make it self-contained, and the job would install them in the
>>>>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <
>>>>>>>>>>> bhulette@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but
>>>>>>>>>>>> it failed again with the same error:
>>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If something comes up with the workers or the tests please
>>>>>>>>>>>>>> reach out to us.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050
>>>>>>>>>>>>>> Zapopan, Jal.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> |
>>>>>>>>>>>>>> Instagram <https://www.instagram.com/wizelineglobal/> |
>>>>>>>>>>>>>> LinkedIn <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> *This email and its contents (including any attachments) are
>>>>>>>>>>>>>> being sent toyou on the condition of confidentiality and may be protected
>>>>>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>>>>> email.*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>
>>>> --
>>>>
>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>
>>>> Site Reliability Engineer III
>>>>
>>>> daniela.martin@wizeline.com
>>>>
>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>
>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>> <https://www.linkedin.com/company/wizeline>
>>>>
>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *This email and its contents (including any attachments) are being sent
>>>> toyou on the condition of confidentiality and may be protected by
>>>> legalprivilege. Access to this email by anyone other than the intended
>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>> immediatelynotify the sender by replying to this message and delete the
>>>> materialimmediately from your system. Any further use, dissemination,
>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>> norepresentation is made with respect to any content contained in this
>>>> email.*
>>>
>>>
>>
>> --
>>
>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>
>> Site Reliability Engineer III
>>
>> daniela.martin@wizeline.com
>>
>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>
>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>> <https://www.facebook.com/WizelineGlobal> | Instagram
>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>> <https://www.linkedin.com/company/wizeline>
>>
>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>
--
Daniela Martín (She/Her) | <https://www.wizeline.com/>
Site Reliability Engineer III
daniela.martin@wizeline.com
Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
<https://www.facebook.com/WizelineGlobal> | Instagram
<https://www.instagram.com/wizelineglobal/> | LinkedIn
<https://www.linkedin.com/company/wizeline>
Share feedback on Clutch <https://clutch.co/review/submit/375119>
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Kiley Sok <ki...@google.com>.
The rest of the workers were updated to the new image and are back online.
Thanks for your patience!
On Mon, Jan 31, 2022 at 4:43 PM Daniela Martín <da...@wizeline.com>
wrote:
> Thanks Kiley, it seems that the failures are not related to the *Ubuntu
> 20.04.3* upgrade.
>
> In case the *Python Release Candidate* job works, we can proceed with the
> upgrade of the remaining Jenkins workers.
>
> Thank you.
>
> Regards,
>
> On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <ki...@google.com> wrote:
>
>> XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
>>
>> On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <
>> daniela.martin@wizeline.com> wrote:
>>
>>> Hi everyone,
>>>
>>> Thank you for the information and details.
>>>
>>> We reviewed the tests and we are taking a look at the following jobs
>>> that started failing close to the date the Jenkins instance was upgraded.
>>> We are not sure if it’s because of the Ubuntu 20.04.3 upgrade or for
>>> another reason.
>>>
>>>
>>> -
>>>
>>> beam_PostCommit_XVR_Direct
>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
>>> -
>>>
>>> beam_PostCommit_XVR_Flink
>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
>>> -
>>>
>>> beam_PostCommit_XVR_Spark3
>>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>>>
>>>
>>> We will continue testing them. Any information regarding these jobs
>>> would be greatly appreciated.
>>>
>>> Emily, could you please re-run the Python Release Candidate
>>> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/> job
>>> in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>>>
>>>
>>> Thank you very much.
>>>
>>> Regards,
>>>
>>> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Hey Daniela and Giomar (& all), do you see any concerns with the
>>>> current image so far? If not, let's upgrade the remaining workers 9-15 to
>>>> the new image. Thanks!
>>>>
>>>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep
>>>>> the pool at 50% capacity with the benched workers still on ubuntu 16 as
>>>>> backup and either upgrade all of the workers to ubuntu 20 or roll back
>>>>> again to ubuntu 16 next week.
>>>>>
>>>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Nexmark failures are from an unrelated change, fyi
>>>>>>
>>>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> It looks like something is resetting the *Restrict where this
>>>>>>> project can be run* label setting on
>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>>>
>>>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>>>> broken by the update:
>>>>>>>
>>>>>>>
>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>>>
>>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>>>
>>>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>>>
>>>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>>>
>>>>>>> I found them by looking at https://ci-beam.apache.org/ and sorting
>>>>>>> by status. Looks like we also have a lot of permared Jenkins suites...
>>>>>>>
>>>>>>> Also, we should maybe think of some sustainable rolling-upgrades for
>>>>>>> Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>>>>>>>
>>>>>>>> I'm also running into these errors on my PR for testing the release
>>>>>>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem
>>>>>>>> to help. Do we have to force the machine per-job/task?
>>>>>>>>
>>>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> Thinking some more, the correct course of action may also be to
>>>>>>>>> fix the job to install necessary Python packages if they are missing. This
>>>>>>>>> would make it self-contained, and the job would install them in the
>>>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>>>
>>>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <
>>>>>>>>>> bhulette@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>>>>>>>> failed again with the same error:
>>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>>>
>>>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If something comes up with the workers or the tests please
>>>>>>>>>>>>> reach out to us.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>>>
>>>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050
>>>>>>>>>>>>> Zapopan, Jal.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *This email and its contents (including any attachments) are
>>>>>>>>>>>>> being sent toyou on the condition of confidentiality and may be protected
>>>>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>>>> email.*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>
>>> --
>>>
>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>
>>> Site Reliability Engineer III
>>>
>>> daniela.martin@wizeline.com
>>>
>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>
>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>> <https://www.linkedin.com/company/wizeline>
>>>
>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>
>
> --
>
> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>
> Site Reliability Engineer III
>
> daniela.martin@wizeline.com
>
> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>
> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
> <https://www.facebook.com/WizelineGlobal> | Instagram
> <https://www.instagram.com/wizelineglobal/> | LinkedIn
> <https://www.linkedin.com/company/wizeline>
>
> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Daniela Martín <da...@wizeline.com>.
Thanks Kiley, it seems that the failures are not related to the *Ubuntu
20.04.3* upgrade.
In case the *Python Release Candidate* job works, we can proceed with the
upgrade of the remaining Jenkins workers.
Thank you.
Regards,
On Mon, Jan 31, 2022 at 6:31 PM Kiley Sok <ki...@google.com> wrote:
> XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
>
> On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <
> daniela.martin@wizeline.com> wrote:
>
>> Hi everyone,
>>
>> Thank you for the information and details.
>>
>> We reviewed the tests and we are taking a look at the following jobs that
>> started failing close to the date the Jenkins instance was upgraded. We are
>> not sure if it’s because of the Ubuntu 20.04.3 upgrade or for another
>> reason.
>>
>>
>> -
>>
>> beam_PostCommit_XVR_Direct
>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
>> -
>>
>> beam_PostCommit_XVR_Flink
>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
>> -
>>
>> beam_PostCommit_XVR_Spark3
>> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>>
>>
>> We will continue testing them. Any information regarding these jobs would
>> be greatly appreciated.
>>
>> Emily, could you please re-run the Python Release Candidate
>> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/> job
>> in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>>
>>
>> Thank you very much.
>>
>> Regards,
>>
>> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Hey Daniela and Giomar (& all), do you see any concerns with the current
>>> image so far? If not, let's upgrade the remaining workers 9-15 to the new
>>> image. Thanks!
>>>
>>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>>
>>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep
>>>> the pool at 50% capacity with the benched workers still on ubuntu 16 as
>>>> backup and either upgrade all of the workers to ubuntu 20 or roll back
>>>> again to ubuntu 16 next week.
>>>>
>>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com> wrote:
>>>>
>>>>> Nexmark failures are from an unrelated change, fyi
>>>>>
>>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> It looks like something is resetting the *Restrict where this
>>>>>> project can be run* label setting on
>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>>
>>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>>> broken by the update:
>>>>>>
>>>>>>
>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>>
>>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>>
>>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>>
>>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>>
>>>>>> I found them by looking at https://ci-beam.apache.org/ and sorting
>>>>>> by status. Looks like we also have a lot of permared Jenkins suites...
>>>>>>
>>>>>> Also, we should maybe think of some sustainable rolling-upgrades for
>>>>>> Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>>>>>>
>>>>>>> I'm also running into these errors on my PR for testing the release
>>>>>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem
>>>>>>> to help. Do we have to force the machine per-job/task?
>>>>>>>
>>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Thinking some more, the correct course of action may also be to fix
>>>>>>>> the job to install necessary Python packages if they are missing. This
>>>>>>>> would make it self-contained, and the job would install them in the
>>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>>
>>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>>
>>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>>>>>>> failed again with the same error:
>>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>>
>>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>>
>>>>>>>>>>>> If something comes up with the workers or the tests please
>>>>>>>>>>>> reach out to us.
>>>>>>>>>>>>
>>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>>
>>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>>
>>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>>
>>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050
>>>>>>>>>>>> Zapopan, Jal.
>>>>>>>>>>>>
>>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>>
>>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *This email and its contents (including any attachments) are
>>>>>>>>>>>> being sent toyou on the condition of confidentiality and may be protected
>>>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>>> email.*
>>>>>>>>>>>
>>>>>>>>>>>
>>
>> --
>>
>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>
>> Site Reliability Engineer III
>>
>> daniela.martin@wizeline.com
>>
>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>
>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>> <https://www.facebook.com/WizelineGlobal> | Instagram
>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>> <https://www.linkedin.com/company/wizeline>
>>
>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>
--
Daniela Martín (She/Her) | <https://www.wizeline.com/>
Site Reliability Engineer III
daniela.martin@wizeline.com
Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
<https://www.facebook.com/WizelineGlobal> | Instagram
<https://www.instagram.com/wizelineglobal/> | LinkedIn
<https://www.linkedin.com/company/wizeline>
Share feedback on Clutch <https://clutch.co/review/submit/375119>
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Kiley Sok <ki...@google.com>.
XVR failures: https://issues.apache.org/jira/browse/BEAM-13778
On Mon, Jan 31, 2022 at 4:02 PM Daniela Martín <da...@wizeline.com>
wrote:
> Hi everyone,
>
> Thank you for the information and details.
>
> We reviewed the tests and we are taking a look at the following jobs that
> started failing close to the date the Jenkins instance was upgraded. We are
> not sure if it’s because of the Ubuntu 20.04.3 upgrade or for another
> reason.
>
>
> -
>
> beam_PostCommit_XVR_Direct
> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
> -
>
> beam_PostCommit_XVR_Flink
> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
> -
>
> beam_PostCommit_XVR_Spark3
> <https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
>
>
> We will continue testing them. Any information regarding these jobs would
> be greatly appreciated.
>
> Emily, could you please re-run the Python Release Candidate
> <https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/> job
> in your PR #16632 <https://github.com/apache/beam/pull/16632>?
>
>
> Thank you very much.
>
> Regards,
>
> On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Hey Daniela and Giomar (& all), do you see any concerns with the current
>> image so far? If not, let's upgrade the remaining workers 9-15 to the new
>> image. Thanks!
>>
>> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep the
>>> pool at 50% capacity with the benched workers still on ubuntu 16 as backup
>>> and either upgrade all of the workers to ubuntu 20 or roll back again to
>>> ubuntu 16 next week.
>>>
>>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com> wrote:
>>>
>>>> Nexmark failures are from an unrelated change, fyi
>>>>
>>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> It looks like something is resetting the *Restrict where this
>>>>> project can be run* label setting on
>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>>> I'll try to rollback all workers to ubuntu 16.
>>>>>
>>>>> Daniela & Giomar: Here are other builds that have likely got
>>>>> broken by the update:
>>>>>
>>>>>
>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>>
>>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>>
>>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>>
>>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>>
>>>>> I found them by looking at https://ci-beam.apache.org/ and sorting
>>>>> by status. Looks like we also have a lot of permared Jenkins suites...
>>>>>
>>>>> Also, we should maybe think of some sustainable rolling-upgrades for
>>>>> Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>>
>>>>>
>>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>>>>>
>>>>>> I'm also running into these errors on my PR for testing the release
>>>>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem
>>>>>> to help. Do we have to force the machine per-job/task?
>>>>>>
>>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Thinking some more, the correct course of action may also be to fix
>>>>>>> the job to install necessary Python packages if they are missing. This
>>>>>>> would make it self-contained, and the job would install them in the
>>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>>
>>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>>
>>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>>>>>> failed again with the same error:
>>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>>
>>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> We are investigating an issue affecting
>>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>>
>>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>>
>>>>>>>>>>> If something comes up with the workers or the tests please reach
>>>>>>>>>>> out to us.
>>>>>>>>>>>
>>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>>
>>>>>>>>>>> Thank you.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>>
>>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>>
>>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>>
>>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan,
>>>>>>>>>>> Jal.
>>>>>>>>>>>
>>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> |
>>>>>>>>>>> Facebook <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>>
>>>>>>>>>>> Share feedback on Clutch
>>>>>>>>>>> <https://clutch.co/review/submit/375119>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *This email and its contents (including any attachments) are
>>>>>>>>>>> being sent toyou on the condition of confidentiality and may be protected
>>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>>> email.*
>>>>>>>>>>
>>>>>>>>>>
>
> --
>
> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>
> Site Reliability Engineer III
>
> daniela.martin@wizeline.com
>
> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>
> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
> <https://www.facebook.com/WizelineGlobal> | Instagram
> <https://www.instagram.com/wizelineglobal/> | LinkedIn
> <https://www.linkedin.com/company/wizeline>
>
> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Daniela Martín <da...@wizeline.com>.
Hi everyone,
Thank you for the information and details.
We reviewed the tests and we are taking a look at the following jobs that
started failing close to the date the Jenkins instance was upgraded. We are
not sure if it’s because of the Ubuntu 20.04.3 upgrade or for another
reason.
-
beam_PostCommit_XVR_Direct
<https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Direct/>
-
beam_PostCommit_XVR_Flink
<https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Flink/>
-
beam_PostCommit_XVR_Spark3
<https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_XVR_Spark3/>
We will continue testing them. Any information regarding these jobs would
be greatly appreciated.
Emily, could you please re-run the Python Release Candidate
<https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/> job in
your PR #16632 <https://github.com/apache/beam/pull/16632>?
Thank you very much.
Regards,
On Mon, Jan 31, 2022 at 4:53 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Hey Daniela and Giomar (& all), do you see any concerns with the current
> image so far? If not, let's upgrade the remaining workers 9-15 to the new
> image. Thanks!
>
> On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep the
>> pool at 50% capacity with the benched workers still on ubuntu 16 as backup
>> and either upgrade all of the workers to ubuntu 20 or roll back again to
>> ubuntu 16 next week.
>>
>> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com> wrote:
>>
>>> Nexmark failures are from an unrelated change, fyi
>>>
>>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> It looks like something is resetting the *Restrict where this project
>>>> can be run* label setting on
>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>>> I'll try to rollback all workers to ubuntu 16.
>>>>
>>>> Daniela & Giomar: Here are other builds that have likely got broken by
>>>> the update:
>>>>
>>>>
>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>>
>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>>
>>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>>
>>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>>
>>>> I found them by looking at https://ci-beam.apache.org/ and sorting by
>>>> status. Looks like we also have a lot of permared Jenkins suites...
>>>>
>>>> Also, we should maybe think of some sustainable rolling-upgrades for
>>>> Jenkins + autorollback to a healthy state as babysitting these upgrades
>>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>>
>>>>
>>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>>>>
>>>>> I'm also running into these errors on my PR for testing the release
>>>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem to
>>>>> help. Do we have to force the machine per-job/task?
>>>>>
>>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Thinking some more, the correct course of action may also be to fix
>>>>>> the job to install necessary Python packages if they are missing. This
>>>>>> would make it self-contained, and the job would install them in the
>>>>>> virtual environment, instead of relying on the global python environment.
>>>>>>
>>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected
>>>>>>> worker #14 that runs ubuntu16. I reconfigured the labels, not sure why my
>>>>>>> changes were not saved initially, and restarted the test on your PR:
>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>>
>>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>>>>> failed again with the same error:
>>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>>
>>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>>> We are investigating an issue affecting
>>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>>
>>>>>>>>
>>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>>
>>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>>> (which is linked in PR description template).
>>>>>>>>>
>>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi everyone,
>>>>>>>>>>
>>>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>>> details on: BEAM-12621
>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>>
>>>>>>>>>> If something comes up with the workers or the tests please reach
>>>>>>>>>> out to us.
>>>>>>>>>>
>>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>>
>>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>>
>>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>>
>>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan,
>>>>>>>>>> Jal.
>>>>>>>>>>
>>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>>>>
>>>>>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *This email and its contents (including any attachments) are
>>>>>>>>>> being sent toyou on the condition of confidentiality and may be protected
>>>>>>>>>> by legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>>> email.*
>>>>>>>>>
>>>>>>>>>
--
Daniela Martín (She/Her) | <https://www.wizeline.com/>
Site Reliability Engineer III
daniela.martin@wizeline.com
Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
<https://www.facebook.com/WizelineGlobal> | Instagram
<https://www.instagram.com/wizelineglobal/> | LinkedIn
<https://www.linkedin.com/company/wizeline>
Share feedback on Clutch <https://clutch.co/review/submit/375119>
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
Hey Daniela and Giomar (& all), do you see any concerns with the current
image so far? If not, let's upgrade the remaining workers 9-15 to the new
image. Thanks!
On Fri, Jan 28, 2022 at 10:05 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep the
> pool at 50% capacity with the benched workers still on ubuntu 16 as backup
> and either upgrade all of the workers to ubuntu 20 or roll back again to
> ubuntu 16 next week.
>
> On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com> wrote:
>
>> Nexmark failures are from an unrelated change, fyi
>>
>> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> It looks like something is resetting the *Restrict where this project
>>> can be run* label setting on
>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>>> I'll try to rollback all workers to ubuntu 16.
>>>
>>> Daniela & Giomar: Here are other builds that have likely got broken by
>>> the update:
>>>
>>>
>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>>
>>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>>
>>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>>
>>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>>
>>> I found them by looking at https://ci-beam.apache.org/ and sorting by
>>> status. Looks like we also have a lot of permared Jenkins suites...
>>>
>>> Also, we should maybe think of some sustainable rolling-upgrades for
>>> Jenkins + autorollback to a healthy state as babysitting these upgrades
>>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>>
>>>
>>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>>>
>>>> I'm also running into these errors on my PR for testing the release
>>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem to
>>>> help. Do we have to force the machine per-job/task?
>>>>
>>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Thinking some more, the correct course of action may also be to fix
>>>>> the job to install necessary Python packages if they are missing. This
>>>>> would make it self-contained, and the job would install them in the
>>>>> virtual environment, instead of relying on the global python environment.
>>>>>
>>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected worker
>>>>>> #14 that runs ubuntu16. I reconfigured the labels, not sure why my changes
>>>>>> were not saved initially, and restarted the test on your PR:
>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>>
>>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>>>> failed again with the same error:
>>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>>
>>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> We are investigating an issue affecting
>>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>>> ubuntu16 nodes for now.
>>>>>>>>
>>>>>>>
>>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>>
>>>>>>>> As a reminder, trigger phrases are in
>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>>> (which is linked in PR description template).
>>>>>>>>
>>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>>> details on: BEAM-12621
>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>>
>>>>>>>>> If something comes up with the workers or the tests please reach
>>>>>>>>> out to us.
>>>>>>>>>
>>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>>
>>>>>>>>> Site Reliability Engineer III
>>>>>>>>>
>>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>>
>>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan,
>>>>>>>>> Jal.
>>>>>>>>>
>>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>>>
>>>>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *This email and its contents (including any attachments) are being
>>>>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>>> email.*
>>>>>>>>
>>>>>>>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
Thanks, Kiley. we are back to Ubuntu 20. For the weekend, we'll keep the
pool at 50% capacity with the benched workers still on ubuntu 16 as backup
and either upgrade all of the workers to ubuntu 20 or roll back again to
ubuntu 16 next week.
On Fri, Jan 28, 2022 at 10:29 AM Kiley Sok <ki...@google.com> wrote:
> Nexmark failures are from an unrelated change, fyi
>
> On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> It looks like something is resetting the *Restrict where this project
>> can be run* label setting on
>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
>> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
>> I'll try to rollback all workers to ubuntu 16.
>>
>> Daniela & Giomar: Here are other builds that have likely got broken by
>> the update:
>>
>>
>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>>
>> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>>
>> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>>
>> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>>
>> I found them by looking at https://ci-beam.apache.org/ and sorting by
>> status. Looks like we also have a lot of permared Jenkins suites...
>>
>> Also, we should maybe think of some sustainable rolling-upgrades for
>> Jenkins + autorollback to a healthy state as babysitting these upgrades
>> becomes quite toilsome. Admittedly, they are not very frequent though.
>>
>>
>> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>>
>>> I'm also running into these errors on my PR for testing the release
>>> https://github.com/apache/beam/pull/16632, re-running doesn't seem to
>>> help. Do we have to force the machine per-job/task?
>>>
>>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Thinking some more, the correct course of action may also be to fix the
>>>> job to install necessary Python packages if they are missing. This would
>>>> make it self-contained, and the job would install them in the virtual
>>>> environment, instead of relying on the global python environment.
>>>>
>>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Thanks, I can see it ran on jenkins woker #4, while I expected worker
>>>>> #14 that runs ubuntu16. I reconfigured the labels, not sure why my changes
>>>>> were not saved initially, and restarted the test on your PR:
>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>>
>>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>>> failed again with the same error:
>>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>>
>>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> We are investigating an issue affecting
>>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>>> ubuntu16 nodes for now.
>>>>>>>
>>>>>>
>>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>>
>>>>>>> As a reminder, trigger phrases are in
>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>>> (which is linked in PR description template).
>>>>>>>
>>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>>> details on: BEAM-12621
>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>>
>>>>>>>> If something comes up with the workers or the tests please reach
>>>>>>>> out to us.
>>>>>>>>
>>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>>
>>>>>>>> Site Reliability Engineer III
>>>>>>>>
>>>>>>>> daniela.martin@wizeline.com
>>>>>>>>
>>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan,
>>>>>>>> Jal.
>>>>>>>>
>>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>>
>>>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *This email and its contents (including any attachments) are being
>>>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>>> email.*
>>>>>>>
>>>>>>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Kiley Sok <ki...@google.com>.
Nexmark failures are from an unrelated change, fyi
On Thu, Jan 27, 2022 at 5:59 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> It looks like something is resetting the *Restrict where this project
> can be run* label setting on
> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
> back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
> I'll try to rollback all workers to ubuntu 16.
>
> Daniela & Giomar: Here are other builds that have likely got broken by
> the update:
>
>
> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
>
> https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
>
> https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
>
> https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
>
> I found them by looking at https://ci-beam.apache.org/ and sorting by
> status. Looks like we also have a lot of permared Jenkins suites...
>
> Also, we should maybe think of some sustainable rolling-upgrades for
> Jenkins + autorollback to a healthy state as babysitting these upgrades
> becomes quite toilsome. Admittedly, they are not very frequent though.
>
>
> On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
>
>> I'm also running into these errors on my PR for testing the release
>> https://github.com/apache/beam/pull/16632, re-running doesn't seem to
>> help. Do we have to force the machine per-job/task?
>>
>> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thinking some more, the correct course of action may also be to fix the
>>> job to install necessary Python packages if they are missing. This would
>>> make it self-contained, and the job would install them in the virtual
>>> environment, instead of relying on the global python environment.
>>>
>>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Thanks, I can see it ran on jenkins woker #4, while I expected worker
>>>> #14 that runs ubuntu16. I reconfigured the labels, not sure why my changes
>>>> were not saved initially, and restarted the test on your PR:
>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>>
>>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it
>>>>> failed again with the same error:
>>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>>
>>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> We are investigating an issue affecting
>>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>>> ubuntu16 nodes for now.
>>>>>>
>>>>>
>>>>>> Aslo, I accidentally canceled several Jenkins jobs in the
>>>>>> build queue, which may have been triggered by your PRs, you many need to
>>>>>> rerun them as well. Sorry about this inconvenience.
>>>>>>
>>>>>> As a reminder, trigger phrases are in
>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>>> (which is linked in PR description template).
>>>>>>
>>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>>> daniela.martin@wizeline.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>>> details on: BEAM-12621
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>>
>>>>>>> If something comes up with the workers or the tests please reach out
>>>>>>> to us.
>>>>>>>
>>>>>>> Please let us know if you have any questions or comments.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>>
>>>>>>> Site Reliability Engineer III
>>>>>>>
>>>>>>> daniela.martin@wizeline.com
>>>>>>>
>>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>>>
>>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>>
>>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *This email and its contents (including any attachments) are being
>>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>>> norepresentation is made with respect to any content contained in this
>>>>>>> email.*
>>>>>>
>>>>>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
It looks like something is resetting the *Restrict where this project can
be run* label setting on
https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/configure
back to "beam" for the jobs that I previously rerouted to beam-ubuntu20.
I'll try to rollback all workers to ubuntu 16.
Daniela & Giomar: Here are other builds that have likely got broken by the
update:
https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2/994/console
https://ci-beam.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java17/
https://ci-beam.apache.org/job/beam_Release_Python_NightlySnapshot/1259/console
https://ci-beam.apache.org/job/beam_PostRelease_Python_Candidate/
I found them by looking at https://ci-beam.apache.org/ and sorting by
status. Looks like we also have a lot of permared Jenkins suites...
Also, we should maybe think of some sustainable rolling-upgrades for
Jenkins + autorollback to a healthy state as babysitting these upgrades
becomes quite toilsome. Admittedly, they are not very frequent though.
On Thu, Jan 27, 2022 at 1:52 PM Emily Ye <em...@google.com> wrote:
> I'm also running into these errors on my PR for testing the release
> https://github.com/apache/beam/pull/16632, re-running doesn't seem to
> help. Do we have to force the machine per-job/task?
>
> On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thinking some more, the correct course of action may also be to fix the
>> job to install necessary Python packages if they are missing. This would
>> make it self-contained, and the job would install them in the virtual
>> environment, instead of relying on the global python environment.
>>
>> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thanks, I can see it ran on jenkins woker #4, while I expected worker
>>> #14 that runs ubuntu16. I reconfigured the labels, not sure why my changes
>>> were not saved initially, and restarted the test on your PR:
>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>>
>>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>>> wrote:
>>>
>>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it failed
>>>> again with the same error:
>>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>>
>>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> We are investigating an issue affecting
>>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>>> ubuntu16 nodes for now.
>>>>>
>>>>
>>>>> Aslo, I accidentally canceled several Jenkins jobs in the build queue,
>>>>> which may have been triggered by your PRs, you many need to rerun them as
>>>>> well. Sorry about this inconvenience.
>>>>>
>>>>> As a reminder, trigger phrases are in
>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>>> (which is linked in PR description template).
>>>>>
>>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>>> daniela.martin@wizeline.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> We are updating the Jenkins workers with a new image that was
>>>>>> upgraded from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more
>>>>>> details on: BEAM-12621
>>>>>> <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>>
>>>>>> If something comes up with the workers or the tests please reach out
>>>>>> to us.
>>>>>>
>>>>>> Please let us know if you have any questions or comments.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>>
>>>>>> Site Reliability Engineer III
>>>>>>
>>>>>> daniela.martin@wizeline.com
>>>>>>
>>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>>
>>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>>
>>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *This email and its contents (including any attachments) are being
>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>> norepresentation is made with respect to any content contained in this
>>>>>> email.*
>>>>>
>>>>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Emily Ye <em...@google.com>.
I'm also running into these errors on my PR for testing the release
https://github.com/apache/beam/pull/16632, re-running doesn't seem to help.
Do we have to force the machine per-job/task?
On Thu, Jan 27, 2022 at 7:42 AM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thinking some more, the correct course of action may also be to fix the
> job to install necessary Python packages if they are missing. This would
> make it self-contained, and the job would install them in the virtual
> environment, instead of relying on the global python environment.
>
> On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks, I can see it ran on jenkins woker #4, while I expected worker #14
>> that runs ubuntu16. I reconfigured the labels, not sure why my changes were
>> not saved initially, and restarted the test on your PR:
>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>>
>> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com>
>> wrote:
>>
>>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it failed
>>> again with the same error:
>>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>>
>>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>>
>>>> We are investigating an issue affecting
>>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>>> ubuntu16 nodes for now.
>>>>
>>>
>>>> Aslo, I accidentally canceled several Jenkins jobs in the build queue,
>>>> which may have been triggered by your PRs, you many need to rerun them as
>>>> well. Sorry about this inconvenience.
>>>>
>>>> As a reminder, trigger phrases are in
>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>>> (which is linked in PR description template).
>>>>
>>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>>> daniela.martin@wizeline.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> We are updating the Jenkins workers with a new image that was upgraded
>>>>> from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more details on:
>>>>> BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>>
>>>>> If something comes up with the workers or the tests please reach out
>>>>> to us.
>>>>>
>>>>> Please let us know if you have any questions or comments.
>>>>>
>>>>> Thank you.
>>>>>
>>>>> Regards,
>>>>>
>>>>> --
>>>>>
>>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>>
>>>>> Site Reliability Engineer III
>>>>>
>>>>> daniela.martin@wizeline.com
>>>>>
>>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>>
>>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>>> <https://www.linkedin.com/company/wizeline>
>>>>>
>>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *This email and its contents (including any attachments) are being
>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>> norepresentation is made with respect to any content contained in this
>>>>> email.*
>>>>
>>>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
Thinking some more, the correct course of action may also be to fix the job
to install necessary Python packages if they are missing. This would make
it self-contained, and the job would install them in the virtual
environment, instead of relying on the global python environment.
On Wed, Jan 26, 2022 at 2:00 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thanks, I can see it ran on jenkins woker #4, while I expected worker #14
> that runs ubuntu16. I reconfigured the labels, not sure why my changes were
> not saved initially, and restarted the test on your PR:
> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
>
> On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com> wrote:
>
>> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it failed
>> again with the same error:
>> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>>
>> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> We are investigating an issue affecting
>>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>>> please rerun it via a trigger phrase, it should be scheduled to run on
>>> ubuntu16 nodes for now.
>>>
>>
>>> Aslo, I accidentally canceled several Jenkins jobs in the build queue,
>>> which may have been triggered by your PRs, you many need to rerun them as
>>> well. Sorry about this inconvenience.
>>>
>>> As a reminder, trigger phrases are in
>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>>> (which is linked in PR description template).
>>>
>>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>>> daniela.martin@wizeline.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> We are updating the Jenkins workers with a new image that was upgraded
>>>> from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more details on:
>>>> BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>
>>>>
>>>> If something comes up with the workers or the tests please reach out to
>>>> us.
>>>>
>>>> Please let us know if you have any questions or comments.
>>>>
>>>> Thank you.
>>>>
>>>> Regards,
>>>>
>>>> --
>>>>
>>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>>
>>>> Site Reliability Engineer III
>>>>
>>>> daniela.martin@wizeline.com
>>>>
>>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>>
>>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>>> <https://www.linkedin.com/company/wizeline>
>>>>
>>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *This email and its contents (including any attachments) are being sent
>>>> toyou on the condition of confidentiality and may be protected by
>>>> legalprivilege. Access to this email by anyone other than the intended
>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>> immediatelynotify the sender by replying to this message and delete the
>>>> materialimmediately from your system. Any further use, dissemination,
>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>> norepresentation is made with respect to any content contained in this
>>>> email.*
>>>
>>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
Thanks, I can see it ran on jenkins woker #4, while I expected worker #14
that runs ubuntu16. I reconfigured the labels, not sure why my changes were
not saved initially, and restarted the test on your PR:
https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/95/
On Wed, Jan 26, 2022 at 1:47 PM Brian Hulette <bh...@google.com> wrote:
> Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it failed
> again with the same error:
> https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
>
> On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> We are investigating an issue affecting
>> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
>> please rerun it via a trigger phrase, it should be scheduled to run on
>> ubuntu16 nodes for now.
>>
>
>> Aslo, I accidentally canceled several Jenkins jobs in the build queue,
>> which may have been triggered by your PRs, you many need to rerun them as
>> well. Sorry about this inconvenience.
>>
>> As a reminder, trigger phrases are in
>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
>> (which is linked in PR description template).
>>
>> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
>> daniela.martin@wizeline.com> wrote:
>>
>>> Hi everyone,
>>>
>>> We are updating the Jenkins workers with a new image that was upgraded
>>> from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more details on:
>>> BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>
>>>
>>> If something comes up with the workers or the tests please reach out to
>>> us.
>>>
>>> Please let us know if you have any questions or comments.
>>>
>>> Thank you.
>>>
>>> Regards,
>>>
>>> --
>>>
>>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>>
>>> Site Reliability Engineer III
>>>
>>> daniela.martin@wizeline.com
>>>
>>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>>
>>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>>> <https://www.facebook.com/WizelineGlobal> | Instagram
>>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>>> <https://www.linkedin.com/company/wizeline>
>>>
>>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Brian Hulette <bh...@google.com>.
Thanks Valentyn. FYI I re-ran a Website_Stage_GCS failure but it failed
again with the same error:
https://ci-beam.apache.org/job/beam_PreCommit_Website_Stage_GCS_Phrase/94/
On Wed, Jan 26, 2022 at 12:13 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> We are investigating an issue affecting
> beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
> please rerun it via a trigger phrase, it should be scheduled to run on
> ubuntu16 nodes for now.
>
> Aslo, I accidentally canceled several Jenkins jobs in the build queue,
> which may have been triggered by your PRs, you many need to rerun them as
> well. Sorry about this inconvenience.
>
> As a reminder, trigger phrases are in
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
> (which is linked in PR description template).
>
> On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <
> daniela.martin@wizeline.com> wrote:
>
>> Hi everyone,
>>
>> We are updating the Jenkins workers with a new image that was upgraded
>> from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more details on:
>> BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>
>>
>> If something comes up with the workers or the tests please reach out to
>> us.
>>
>> Please let us know if you have any questions or comments.
>>
>> Thank you.
>>
>> Regards,
>>
>> --
>>
>> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>>
>> Site Reliability Engineer III
>>
>> daniela.martin@wizeline.com
>>
>> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>>
>> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
>> <https://www.facebook.com/WizelineGlobal> | Instagram
>> <https://www.instagram.com/wizelineglobal/> | LinkedIn
>> <https://www.linkedin.com/company/wizeline>
>>
>> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>
Re: Upgrading Jenkins Workers BEAM-12621
Posted by Valentyn Tymofieiev <va...@google.com>.
We are investigating an issue affecting
beam_PreCommit_Website_Stage_GCS_Commit jobs. If this job failed for you,
please rerun it via a trigger phrase, it should be scheduled to run on
ubuntu16 nodes for now.
Aslo, I accidentally canceled several Jenkins jobs in the build queue,
which may have been triggered by your PRs, you many need to rerun them as
well. Sorry about this inconvenience.
As a reminder, trigger phrases are in
https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
(which is linked in PR description template).
On Wed, Jan 26, 2022 at 10:30 AM Daniela Martín <da...@wizeline.com>
wrote:
> Hi everyone,
>
> We are updating the Jenkins workers with a new image that was upgraded
> from Ubuntu 16.0 LTS to Ubuntu 20.04.3 LTS. You can see more details on:
> BEAM-12621 <https://issues.apache.org/jira/browse/BEAM-12621>
>
> If something comes up with the workers or the tests please reach out to us.
>
> Please let us know if you have any questions or comments.
>
> Thank you.
>
> Regards,
>
> --
>
> Daniela Martín (She/Her) | <https://www.wizeline.com/>
>
> Site Reliability Engineer III
>
> daniela.martin@wizeline.com
>
> Amado Nervo 2200, Esfera P6, Col. Ciudad del Sol, 45050 Zapopan, Jal.
>
> Follow us Twitter <https://twitter.com/wizelineglobal> | Facebook
> <https://www.facebook.com/WizelineGlobal> | Instagram
> <https://www.instagram.com/wizelineglobal/> | LinkedIn
> <https://www.linkedin.com/company/wizeline>
>
> Share feedback on Clutch <https://clutch.co/review/submit/375119>
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*