Re: Update

[Jun HE <ju...@apache.org>]

Thanks a lot for the update, Olaf!

Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：

> Hi,
>
> All machines patched. Jenkins and it plugins are updated:
>
> Things to be noted:
>
> * Slave 2 seems to be in serious problems. The disk image seems to be
> corrupt, I would say:
> One of the problems: docker does not start any more.
> Is there anything important on it ? If yes please contact me. I would
> recommend to set up slave2 from scratch again.
>
> * There was a warning regarding Copy Artifacts Plugin. It now imposes
> stricter rules. Not sure if there is a job depending on it.
>
> * I removed the CVS plugin.
>
> Everything else seem to working as usual.
>
> Best,
> Olaf
>
>
>
>
> > Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> >
> > Hi,
> >
> > I am doing an update of the machines in CI . Seems a couple of security
> fixes are to be applied.
> >
> > Olaf
>
>

Re: Update

[Evans Ye <ev...@apache.org>]

Looks awesome! Thank you Kengo. I think this is an important change that
can further support dev efficiency and enable us to release early,  release
often.

Kengo Seki <se...@apache.org> 於 2020年11月19日 週四 上午8:23寫道：

> Thank you for the comment, Evans and Olaf! Following your advices, I
> did the following:
>
> * Removed unused EBS volumes (1TBx2 and 30GBx1).
> * Replaced slaves (02, 03, 06, 07) with newly created EC2 instances.
> Instance types were also upgraded (m3, m4 -> m5).
> * Attached EBS volumes to the above instances. 200GBs to 02 and 03,
> 800GB to 06 and 07.
>   (I said that I was going to separate two 2TB volumes into four
> 500GBs in the past, but slave 06 had used 660GB+ before replacing, so
> I changed the allocation)
>
> Then we're using the following resources now, in accordance with Evans'
> email:
>
> * EC2 instances: one m3.xlarge (master), three m5.xlarge (slave 02, 03
> and 07) and one m5.2xlarge (slave 06)
> * EBS volumes: one 1TB gp2 (master), two 200GB gp2 (slave 02 and 03),
> and two 800GB gp2 (slave 06 and 07)
>
> New servers seem to work fine with Jenkins [1]. I also updated the
> "Bigtop CI Setup Guide" page on cwiki [2].
> Let me know if you find something wrong! :)
>
> [1]: https://ci.bigtop.apache.org/computer/
> [2]:
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide#BigtopCISetupGuide-SetupJenkinsslaves
>
> Kengo Seki <se...@apache.org>
>
> On Wed, Nov 11, 2020 at 4:59 AM Olaf Flebbe <of...@oflebbe.de> wrote:
> >
> > hi,
> >
> > fully supporting evans:
> > the unconnected disk do not contain anything valuable, please remove. it
> might make sense to even recreate the current disks on ssd, a bit larger as
> before if needed.
> >
> > olaf
> >
> > > Am 10.11.2020 um 08:09 schrieb Evans Ye <ev...@apache.org>:
> > >
> > > Yes I think overall your plan is good.
> > > What's the purpose of leveraging EBS snapshot? Is it to backup the
> things
> > > we have before migration?
> > > Except for the master node(have jenkins settings stored on disk), all
> those
> > > slaves can be wiped out directly.
> > >
> > >
> > >
> > > Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午2:42寫道：
> > >
> > >> Thanks everyone for the information! Now I understand our
> circumstances.
> > >> So we're going to split two 1TB volumes attached to slave06 and 07
> > >> into four 500GB volumes (and change their type to gp2), reattach them
> > >> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> > >> right?
> > >>
> > >>> Kengo would you like to take this, or you need a help?
> > >>
> > >> I think I can do them somehow (maybe using EBS snapshot?), but let me
> > >> ask your help if I'm stuck. :)
> > >>
> > >> Kengo Seki <se...@apache.org>
> > >>
> > >> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
> > >>>
> > >>> OK. I got it now.
> > >>> So the newly created volumes are currently attached to slave06_2 and
> > >>> slave07_2, respectively.
> > >>> However, they're standard HDD, not GP2 SSD. I think we can take this
> > >> chance
> > >>> to recreate those 2 slaves and do an overhaul of our infrastructure.
> > >>>
> > >>> Kengo would you like to take this, or you need a help?
> > >>>
> > >>> Evans
> > >>>
> > >>> Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> OMG . I think I did it.
> > >>>>
> > >>>> A few years ago two of the instance had a hardware problems and did
> not
> > >>>> reboot any more, filesystem was corrupted and so on.  That was at
> the
> > >> time
> > >>>> of the spectre vulnarability discovery. (2018) . At that time AWS
> had
> > >> major
> > >>>> instabilities since updating firmware seem to have failed for some
> > >> classes
> > >>>> of hardware.
> > >>>>
> > >>>> I tried to recreate them as close as possible but I may have left
> > >>>> accidentely the volumes around. Please lets delete them.
> > >>>>
> > >>>> Olaf
> > >>>>
> > >>>>> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <cos@apache.org
> >:
> > >>>>>
> > >>>>> Thanks Evans!
> > >>>>>
> > >>>>> It's great you found the details: they are definitely accurate as I
> > >> am
> > >>>>> recalling now. Kengo, do you think splitting the volumes would help
> > >> us
> > >>>> for a
> > >>>>> while? Or perhaps we shall try to expand the resource pool (which
> > >> might
> > >>>> take a
> > >>>>> while)?
> > >>>>>
> > >>>>> Thanks!
> > >>>>> Cos
> > >>>>>
> > >>>>> On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > >>>>>> In fact, the original deal of our resource is as follows:
> > >>>>>>
> > >>>>>>> 1 m3.2xlarge for CI
> > >>>>>>> 4 m3.xlarge for CI and demo
> > >>>>>>> 3 1TB EBS volumes
> > >>>>>>> 5 elastic IP addresses
> > >>>>>>
> > >>>>>> So technically we should not use that 2 additional 1T volumes
> > >> (created
> > >>>> in
> > >>>>>> 2018).
> > >>>>>> Instead, I think what we can do is to split up one of the existing
> > >> 1TB
> > >>>>>> volumes(ex: attached to slave07) into smaller volumes for slave02,
> > >> 03.
> > >>>>>>
> > >>>>>>
> > >>>>>> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> > >>>>>>
> > >>>>>>> Kengo,
> > >>>>>>>
> > >>>>>>> We had an agreement with EMR folks that we are using the
> resources
> > >>>>>>> available
> > >>>>>>> to us and it is included into their budget (or something to this
> > >>>> extent).
> > >>>>>>> If
> > >>>>>>> you see some of the resources available under our account - I
> > >> don't see
> > >>>>>>> why we
> > >>>>>>> can't use them.
> > >>>>>>>
> > >>>>>>> If for whatever reason we need to expand the pool, that would
> > >> require a
> > >>>>>>> separate conversation with nice folks from that team, I imagine.
> > >> Please
> > >>>>>>> let me
> > >>>>>>> know if I can help with this going forward.
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>> Cos
> > >>>>>>>
> > >>>>>>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > >>>>>>>> Thanks for the comment, Cos! I was able to start docker service
> on
> > >>>>>>>> docker-slave-02 without replacing and am running some Jenkins
> > >> jobs on
> > >>>>>>>> it now, so I'll replace it in the short future.
> > >>>>>>>> I have a few things that I'd like to ask additionally:
> > >>>>>>>>
> > >>>>>>>> * docker-slave-02 and 03 have a gp2 storage as a root volume
> that
> > >> has
> > >>>>>>>> only 8GiB capacity, and they sometimes run short and stop the
> CI.
> > >>>>>>>> May I increase them to 20 or 30 GiB when I replace those
> > >> instances?
> > >>>>>>>> (I'm not sure what is our budget)
> > >>>>>>>>
> > >>>>>>>> * They use an instance store with 30GiB to put docker images
> into
> > >> it,
> > >>>>>>>> and they also sometimes run short.
> > >>>>>>>> It seems there are two unused volumes with 1TiB (vol-ae71114e
> and
> > >>>>>>>> vol-4efa69ae) on AWS console.
> > >>>>>>>> May I attach them to 02 and 03 instead of instance stores, or
> are
> > >>>>>>>> they backups or something?
> > >>>>>>>>
> > >>>>>>>> Kengo Seki <se...@apache.org>
> > >>>>>>>>
> > >>>>>>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <
> cos@apache.org
> > >>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> I'd say let replace the broken one. I don't think there's a
> > >>>> sentimental
> > >>>>>>>>> value attached ;)
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> With regards,
> > >>>>>>>>>  Cos
> > >>>>>>>>>
> > >>>>>>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> > >>>>>>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI
> > >> became
> > >>>>>>> cool :)
> > >>>>>>>>>> Regarding docker-slave-02, I'll try to replace it after
> waiting
> > >> for
> > >>>> a
> > >>>>>>>>>> while to make sure there's no objection.
> > >>>>>>>>>>
> > >>>>>>>>>> Kengo Seki <se...@apache.org>
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org>
> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks a lot for the update, Olaf!
> > >>>>>>>>>>>
> > >>>>>>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> All machines patched. Jenkins and it plugins are updated:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Things to be noted:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * Slave 2 seems to be in serious problems. The disk image
> > >> seems to
> > >>>>>>> be
> > >>>>>>>>>>>> corrupt, I would say:
> > >>>>>>>>>>>> One of the problems: docker does not start any more.
> > >>>>>>>>>>>> Is there anything important on it ? If yes please contact
> me.
> > >> I
> > >>>>>>> would
> > >>>>>>>>>>>> recommend to set up slave2 from scratch again.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It
> now
> > >>>>>>> imposes
> > >>>>>>>>>>>> stricter rules. Not sure if there is a job depending on it.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> * I removed the CVS plugin.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Everything else seem to working as usual.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Olaf
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of@oflebbe.de
> >:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I am doing an update of the machines in CI . Seems a couple
> > >> of
> > >>>>>>> security
> > >>>>>>>>>>>> fixes are to be applied.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Olaf
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>
> > >>>>
> > >>>>
> > >>
> >
>

Re: Update

[Kengo Seki <se...@apache.org>]

Thank you for the comment, Evans and Olaf! Following your advices, I
did the following:

* Removed unused EBS volumes (1TBx2 and 30GBx1).
* Replaced slaves (02, 03, 06, 07) with newly created EC2 instances.
Instance types were also upgraded (m3, m4 -> m5).
* Attached EBS volumes to the above instances. 200GBs to 02 and 03,
800GB to 06 and 07.
  (I said that I was going to separate two 2TB volumes into four
500GBs in the past, but slave 06 had used 660GB+ before replacing, so
I changed the allocation)

Then we're using the following resources now, in accordance with Evans' email:

* EC2 instances: one m3.xlarge (master), three m5.xlarge (slave 02, 03
and 07) and one m5.2xlarge (slave 06)
* EBS volumes: one 1TB gp2 (master), two 200GB gp2 (slave 02 and 03),
and two 800GB gp2 (slave 06 and 07)

New servers seem to work fine with Jenkins [1]. I also updated the
"Bigtop CI Setup Guide" page on cwiki [2].
Let me know if you find something wrong! :)

[1]: https://ci.bigtop.apache.org/computer/
[2]: https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide#BigtopCISetupGuide-SetupJenkinsslaves

Kengo Seki <se...@apache.org>

On Wed, Nov 11, 2020 at 4:59 AM Olaf Flebbe <of...@oflebbe.de> wrote:
>
> hi,
>
> fully supporting evans:
> the unconnected disk do not contain anything valuable, please remove. it might make sense to even recreate the current disks on ssd, a bit larger as before if needed.
>
> olaf
>
> > Am 10.11.2020 um 08:09 schrieb Evans Ye <ev...@apache.org>:
> >
> > Yes I think overall your plan is good.
> > What's the purpose of leveraging EBS snapshot? Is it to backup the things
> > we have before migration?
> > Except for the master node(have jenkins settings stored on disk), all those
> > slaves can be wiped out directly.
> >
> >
> >
> > Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午2:42寫道：
> >
> >> Thanks everyone for the information! Now I understand our circumstances.
> >> So we're going to split two 1TB volumes attached to slave06 and 07
> >> into four 500GB volumes (and change their type to gp2), reattach them
> >> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> >> right?
> >>
> >>> Kengo would you like to take this, or you need a help?
> >>
> >> I think I can do them somehow (maybe using EBS snapshot?), but let me
> >> ask your help if I'm stuck. :)
> >>
> >> Kengo Seki <se...@apache.org>
> >>
> >> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
> >>>
> >>> OK. I got it now.
> >>> So the newly created volumes are currently attached to slave06_2 and
> >>> slave07_2, respectively.
> >>> However, they're standard HDD, not GP2 SSD. I think we can take this
> >> chance
> >>> to recreate those 2 slaves and do an overhaul of our infrastructure.
> >>>
> >>> Kengo would you like to take this, or you need a help?
> >>>
> >>> Evans
> >>>
> >>> Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
> >>>
> >>>> Hi,
> >>>>
> >>>> OMG . I think I did it.
> >>>>
> >>>> A few years ago two of the instance had a hardware problems and did not
> >>>> reboot any more, filesystem was corrupted and so on.  That was at the
> >> time
> >>>> of the spectre vulnarability discovery. (2018) . At that time AWS had
> >> major
> >>>> instabilities since updating firmware seem to have failed for some
> >> classes
> >>>> of hardware.
> >>>>
> >>>> I tried to recreate them as close as possible but I may have left
> >>>> accidentely the volumes around. Please lets delete them.
> >>>>
> >>>> Olaf
> >>>>
> >>>>> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
> >>>>>
> >>>>> Thanks Evans!
> >>>>>
> >>>>> It's great you found the details: they are definitely accurate as I
> >> am
> >>>>> recalling now. Kengo, do you think splitting the volumes would help
> >> us
> >>>> for a
> >>>>> while? Or perhaps we shall try to expand the resource pool (which
> >> might
> >>>> take a
> >>>>> while)?
> >>>>>
> >>>>> Thanks!
> >>>>> Cos
> >>>>>
> >>>>> On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> >>>>>> In fact, the original deal of our resource is as follows:
> >>>>>>
> >>>>>>> 1 m3.2xlarge for CI
> >>>>>>> 4 m3.xlarge for CI and demo
> >>>>>>> 3 1TB EBS volumes
> >>>>>>> 5 elastic IP addresses
> >>>>>>
> >>>>>> So technically we should not use that 2 additional 1T volumes
> >> (created
> >>>> in
> >>>>>> 2018).
> >>>>>> Instead, I think what we can do is to split up one of the existing
> >> 1TB
> >>>>>> volumes(ex: attached to slave07) into smaller volumes for slave02,
> >> 03.
> >>>>>>
> >>>>>>
> >>>>>> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> >>>>>>
> >>>>>>> Kengo,
> >>>>>>>
> >>>>>>> We had an agreement with EMR folks that we are using the resources
> >>>>>>> available
> >>>>>>> to us and it is included into their budget (or something to this
> >>>> extent).
> >>>>>>> If
> >>>>>>> you see some of the resources available under our account - I
> >> don't see
> >>>>>>> why we
> >>>>>>> can't use them.
> >>>>>>>
> >>>>>>> If for whatever reason we need to expand the pool, that would
> >> require a
> >>>>>>> separate conversation with nice folks from that team, I imagine.
> >> Please
> >>>>>>> let me
> >>>>>>> know if I can help with this going forward.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>> Cos
> >>>>>>>
> >>>>>>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> >>>>>>>> Thanks for the comment, Cos! I was able to start docker service on
> >>>>>>>> docker-slave-02 without replacing and am running some Jenkins
> >> jobs on
> >>>>>>>> it now, so I'll replace it in the short future.
> >>>>>>>> I have a few things that I'd like to ask additionally:
> >>>>>>>>
> >>>>>>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that
> >> has
> >>>>>>>> only 8GiB capacity, and they sometimes run short and stop the CI.
> >>>>>>>> May I increase them to 20 or 30 GiB when I replace those
> >> instances?
> >>>>>>>> (I'm not sure what is our budget)
> >>>>>>>>
> >>>>>>>> * They use an instance store with 30GiB to put docker images into
> >> it,
> >>>>>>>> and they also sometimes run short.
> >>>>>>>> It seems there are two unused volumes with 1TiB (vol-ae71114e and
> >>>>>>>> vol-4efa69ae) on AWS console.
> >>>>>>>> May I attach them to 02 and 03 instead of instance stores, or are
> >>>>>>>> they backups or something?
> >>>>>>>>
> >>>>>>>> Kengo Seki <se...@apache.org>
> >>>>>>>>
> >>>>>>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <cos@apache.org
> >>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> I'd say let replace the broken one. I don't think there's a
> >>>> sentimental
> >>>>>>>>> value attached ;)
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> With regards,
> >>>>>>>>>  Cos
> >>>>>>>>>
> >>>>>>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> >>>>>>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI
> >> became
> >>>>>>> cool :)
> >>>>>>>>>> Regarding docker-slave-02, I'll try to replace it after waiting
> >> for
> >>>> a
> >>>>>>>>>> while to make sure there's no objection.
> >>>>>>>>>>
> >>>>>>>>>> Kengo Seki <se...@apache.org>
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks a lot for the update, Olaf!
> >>>>>>>>>>>
> >>>>>>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> All machines patched. Jenkins and it plugins are updated:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Things to be noted:
> >>>>>>>>>>>>
> >>>>>>>>>>>> * Slave 2 seems to be in serious problems. The disk image
> >> seems to
> >>>>>>> be
> >>>>>>>>>>>> corrupt, I would say:
> >>>>>>>>>>>> One of the problems: docker does not start any more.
> >>>>>>>>>>>> Is there anything important on it ? If yes please contact me.
> >> I
> >>>>>>> would
> >>>>>>>>>>>> recommend to set up slave2 from scratch again.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
> >>>>>>> imposes
> >>>>>>>>>>>> stricter rules. Not sure if there is a job depending on it.
> >>>>>>>>>>>>
> >>>>>>>>>>>> * I removed the CVS plugin.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Everything else seem to working as usual.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Olaf
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I am doing an update of the machines in CI . Seems a couple
> >> of
> >>>>>>> security
> >>>>>>>>>>>> fixes are to be applied.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Olaf
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>
> >>>>
> >>
>

Re: Update

[Olaf Flebbe <of...@oflebbe.de>]

hi,

fully supporting evans:
the unconnected disk do not contain anything valuable, please remove. it might make sense to even recreate the current disks on ssd, a bit larger as before if needed.

olaf

> Am 10.11.2020 um 08:09 schrieb Evans Ye <ev...@apache.org>:
> 
> Yes I think overall your plan is good.
> What's the purpose of leveraging EBS snapshot? Is it to backup the things
> we have before migration?
> Except for the master node(have jenkins settings stored on disk), all those
> slaves can be wiped out directly.
> 
> 
> 
> Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午2:42寫道：
> 
>> Thanks everyone for the information! Now I understand our circumstances.
>> So we're going to split two 1TB volumes attached to slave06 and 07
>> into four 500GB volumes (and change their type to gp2), reattach them
>> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
>> right?
>> 
>>> Kengo would you like to take this, or you need a help?
>> 
>> I think I can do them somehow (maybe using EBS snapshot?), but let me
>> ask your help if I'm stuck. :)
>> 
>> Kengo Seki <se...@apache.org>
>> 
>> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
>>> 
>>> OK. I got it now.
>>> So the newly created volumes are currently attached to slave06_2 and
>>> slave07_2, respectively.
>>> However, they're standard HDD, not GP2 SSD. I think we can take this
>> chance
>>> to recreate those 2 slaves and do an overhaul of our infrastructure.
>>> 
>>> Kengo would you like to take this, or you need a help?
>>> 
>>> Evans
>>> 
>>> Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
>>> 
>>>> Hi,
>>>> 
>>>> OMG . I think I did it.
>>>> 
>>>> A few years ago two of the instance had a hardware problems and did not
>>>> reboot any more, filesystem was corrupted and so on.  That was at the
>> time
>>>> of the spectre vulnarability discovery. (2018) . At that time AWS had
>> major
>>>> instabilities since updating firmware seem to have failed for some
>> classes
>>>> of hardware.
>>>> 
>>>> I tried to recreate them as close as possible but I may have left
>>>> accidentely the volumes around. Please lets delete them.
>>>> 
>>>> Olaf
>>>> 
>>>>> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
>>>>> 
>>>>> Thanks Evans!
>>>>> 
>>>>> It's great you found the details: they are definitely accurate as I
>> am
>>>>> recalling now. Kengo, do you think splitting the volumes would help
>> us
>>>> for a
>>>>> while? Or perhaps we shall try to expand the resource pool (which
>> might
>>>> take a
>>>>> while)?
>>>>> 
>>>>> Thanks!
>>>>> Cos
>>>>> 
>>>>> On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
>>>>>> In fact, the original deal of our resource is as follows:
>>>>>> 
>>>>>>> 1 m3.2xlarge for CI
>>>>>>> 4 m3.xlarge for CI and demo
>>>>>>> 3 1TB EBS volumes
>>>>>>> 5 elastic IP addresses
>>>>>> 
>>>>>> So technically we should not use that 2 additional 1T volumes
>> (created
>>>> in
>>>>>> 2018).
>>>>>> Instead, I think what we can do is to split up one of the existing
>> 1TB
>>>>>> volumes(ex: attached to slave07) into smaller volumes for slave02,
>> 03.
>>>>>> 
>>>>>> 
>>>>>> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
>>>>>> 
>>>>>>> Kengo,
>>>>>>> 
>>>>>>> We had an agreement with EMR folks that we are using the resources
>>>>>>> available
>>>>>>> to us and it is included into their budget (or something to this
>>>> extent).
>>>>>>> If
>>>>>>> you see some of the resources available under our account - I
>> don't see
>>>>>>> why we
>>>>>>> can't use them.
>>>>>>> 
>>>>>>> If for whatever reason we need to expand the pool, that would
>> require a
>>>>>>> separate conversation with nice folks from that team, I imagine.
>> Please
>>>>>>> let me
>>>>>>> know if I can help with this going forward.
>>>>>>> 
>>>>>>> Thanks!
>>>>>>> Cos
>>>>>>> 
>>>>>>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
>>>>>>>> Thanks for the comment, Cos! I was able to start docker service on
>>>>>>>> docker-slave-02 without replacing and am running some Jenkins
>> jobs on
>>>>>>>> it now, so I'll replace it in the short future.
>>>>>>>> I have a few things that I'd like to ask additionally:
>>>>>>>> 
>>>>>>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that
>> has
>>>>>>>> only 8GiB capacity, and they sometimes run short and stop the CI.
>>>>>>>> May I increase them to 20 or 30 GiB when I replace those
>> instances?
>>>>>>>> (I'm not sure what is our budget)
>>>>>>>> 
>>>>>>>> * They use an instance store with 30GiB to put docker images into
>> it,
>>>>>>>> and they also sometimes run short.
>>>>>>>> It seems there are two unused volumes with 1TiB (vol-ae71114e and
>>>>>>>> vol-4efa69ae) on AWS console.
>>>>>>>> May I attach them to 02 and 03 instead of instance stores, or are
>>>>>>>> they backups or something?
>>>>>>>> 
>>>>>>>> Kengo Seki <se...@apache.org>
>>>>>>>> 
>>>>>>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <cos@apache.org
>>> 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> I'd say let replace the broken one. I don't think there's a
>>>> sentimental
>>>>>>>>> value attached ;)
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> With regards,
>>>>>>>>>  Cos
>>>>>>>>> 
>>>>>>>>> On 02.11.2020 08:16, Kengo Seki wrote:
>>>>>>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI
>> became
>>>>>>> cool :)
>>>>>>>>>> Regarding docker-slave-02, I'll try to replace it after waiting
>> for
>>>> a
>>>>>>>>>> while to make sure there's no objection.
>>>>>>>>>> 
>>>>>>>>>> Kengo Seki <se...@apache.org>
>>>>>>>>>> 
>>>>>>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Thanks a lot for the update, Olaf!
>>>>>>>>>>> 
>>>>>>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> All machines patched. Jenkins and it plugins are updated:
>>>>>>>>>>>> 
>>>>>>>>>>>> Things to be noted:
>>>>>>>>>>>> 
>>>>>>>>>>>> * Slave 2 seems to be in serious problems. The disk image
>> seems to
>>>>>>> be
>>>>>>>>>>>> corrupt, I would say:
>>>>>>>>>>>> One of the problems: docker does not start any more.
>>>>>>>>>>>> Is there anything important on it ? If yes please contact me.
>> I
>>>>>>> would
>>>>>>>>>>>> recommend to set up slave2 from scratch again.
>>>>>>>>>>>> 
>>>>>>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
>>>>>>> imposes
>>>>>>>>>>>> stricter rules. Not sure if there is a job depending on it.
>>>>>>>>>>>> 
>>>>>>>>>>>> * I removed the CVS plugin.
>>>>>>>>>>>> 
>>>>>>>>>>>> Everything else seem to working as usual.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Olaf
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I am doing an update of the machines in CI . Seems a couple
>> of
>>>>>>> security
>>>>>>>>>>>> fixes are to be applied.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Olaf
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>> 
>>>> 
>> 

Re: Update

[Evans Ye <ev...@apache.org>]

Yes. I think that's enough.
Following is what I was used for bootstrapping a slave w/o volume, you can
take as a REF and modify from there:

adduser jenkins
mkdir /media/ephemeral0/docker
ln -s /media/ephemeral0/docker /var/lib/
yum install -y docker git java ruby
amazon-linux-extras install -y docker
usermod -a -G docker jenkins
mkdir /home/jenkins/.ssh
cat << EOF > /home/jenkins/.ssh/authorized_keys
MASTER_PUBLIC_KEY_HERE
EOF
chmod 700 /home/jenkins/.ssh
chmod 400 /home/jenkins/.ssh/authorized_keys
chown -R jenkins:jenkins /home/jenkins
service docker start
curl -L "
https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname
-s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose


Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午4:31寫道：

> I intended to keep current data in the storage and only shrink its
> capacity, because I thought slaves also have some settings on the
> disk.
> But are following steps enough for slave setup? If so, I'll wipe and
> setup them from scratch.
>
> * launch EC2 instance
> * create jenkins user
> * configure its authorized_keys so that master can login with current key
> * change slave's IP address on master
> * (then master automatically configure this node, e.g., installing
> agent, right?)
>
> Kengo Seki <se...@apache.org>
>
> On Tue, Nov 10, 2020 at 4:09 PM Evans Ye <ev...@apache.org> wrote:
> >
> > Yes I think overall your plan is good.
> > What's the purpose of leveraging EBS snapshot? Is it to backup the things
> > we have before migration?
> > Except for the master node(have jenkins settings stored on disk), all
> those
> > slaves can be wiped out directly.
> >
> >
> >
> > Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午2:42寫道：
> >
> > > Thanks everyone for the information! Now I understand our
> circumstances.
> > > So we're going to split two 1TB volumes attached to slave06 and 07
> > > into four 500GB volumes (and change their type to gp2), reattach them
> > > to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> > > right?
> > >
> > > > Kengo would you like to take this, or you need a help?
> > >
> > > I think I can do them somehow (maybe using EBS snapshot?), but let me
> > > ask your help if I'm stuck. :)
> > >
> > > Kengo Seki <se...@apache.org>
> > >
> > > On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
> > > >
> > > > OK. I got it now.
> > > > So the newly created volumes are currently attached to slave06_2 and
> > > > slave07_2, respectively.
> > > > However, they're standard HDD, not GP2 SSD. I think we can take this
> > > chance
> > > > to recreate those 2 slaves and do an overhaul of our infrastructure.
> > > >
> > > > Kengo would you like to take this, or you need a help?
> > > >
> > > > Evans
> > > >
> > > > Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
> > > >
> > > > > Hi,
> > > > >
> > > > > OMG . I think I did it.
> > > > >
> > > > > A few years ago two of the instance had a hardware problems and
> did not
> > > > > reboot any more, filesystem was corrupted and so on.  That was at
> the
> > > time
> > > > > of the spectre vulnarability discovery. (2018) . At that time AWS
> had
> > > major
> > > > > instabilities since updating firmware seem to have failed for some
> > > classes
> > > > > of hardware.
> > > > >
> > > > > I tried to recreate them as close as possible but I may have left
> > > > > accidentely the volumes around. Please lets delete them.
> > > > >
> > > > > Olaf
> > > > >
> > > > > > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <
> cos@apache.org>:
> > > > > >
> > > > > > Thanks Evans!
> > > > > >
> > > > > > It's great you found the details: they are definitely accurate
> as I
> > > am
> > > > > > recalling now. Kengo, do you think splitting the volumes would
> help
> > > us
> > > > > for a
> > > > > > while? Or perhaps we shall try to expand the resource pool (which
> > > might
> > > > > take a
> > > > > > while)?
> > > > > >
> > > > > > Thanks!
> > > > > >  Cos
> > > > > >
> > > > > > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > > > > >> In fact, the original deal of our resource is as follows:
> > > > > >>
> > > > > >>> 1 m3.2xlarge for CI
> > > > > >>> 4 m3.xlarge for CI and demo
> > > > > >>> 3 1TB EBS volumes
> > > > > >>> 5 elastic IP addresses
> > > > > >>
> > > > > >> So technically we should not use that 2 additional 1T volumes
> > > (created
> > > > > in
> > > > > >> 2018).
> > > > > >> Instead, I think what we can do is to split up one of the
> existing
> > > 1TB
> > > > > >> volumes(ex: attached to slave07) into smaller volumes for
> slave02,
> > > 03.
> > > > > >>
> > > > > >>
> > > > > >> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> > > > > >>
> > > > > >>> Kengo,
> > > > > >>>
> > > > > >>> We had an agreement with EMR folks that we are using the
> resources
> > > > > >>> available
> > > > > >>> to us and it is included into their budget (or something to
> this
> > > > > extent).
> > > > > >>> If
> > > > > >>> you see some of the resources available under our account - I
> > > don't see
> > > > > >>> why we
> > > > > >>> can't use them.
> > > > > >>>
> > > > > >>> If for whatever reason we need to expand the pool, that would
> > > require a
> > > > > >>> separate conversation with nice folks from that team, I
> imagine.
> > > Please
> > > > > >>> let me
> > > > > >>> know if I can help with this going forward.
> > > > > >>>
> > > > > >>> Thanks!
> > > > > >>>  Cos
> > > > > >>>
> > > > > >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > > > > >>>> Thanks for the comment, Cos! I was able to start docker
> service on
> > > > > >>>> docker-slave-02 without replacing and am running some Jenkins
> > > jobs on
> > > > > >>>> it now, so I'll replace it in the short future.
> > > > > >>>> I have a few things that I'd like to ask additionally:
> > > > > >>>>
> > > > > >>>> * docker-slave-02 and 03 have a gp2 storage as a root volume
> that
> > > has
> > > > > >>>> only 8GiB capacity, and they sometimes run short and stop the
> CI.
> > > > > >>>>  May I increase them to 20 or 30 GiB when I replace those
> > > instances?
> > > > > >>>> (I'm not sure what is our budget)
> > > > > >>>>
> > > > > >>>> * They use an instance store with 30GiB to put docker images
> into
> > > it,
> > > > > >>>> and they also sometimes run short.
> > > > > >>>>  It seems there are two unused volumes with 1TiB
> (vol-ae71114e and
> > > > > >>>> vol-4efa69ae) on AWS console.
> > > > > >>>>  May I attach them to 02 and 03 instead of instance stores,
> or are
> > > > > >>>> they backups or something?
> > > > > >>>>
> > > > > >>>> Kengo Seki <se...@apache.org>
> > > > > >>>>
> > > > > >>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <
> cos@apache.org
> > > >
> > > > > >>> wrote:
> > > > > >>>>>
> > > > > >>>>> I'd say let replace the broken one. I don't think there's a
> > > > > sentimental
> > > > > >>>>> value attached ;)
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> With regards,
> > > > > >>>>>   Cos
> > > > > >>>>>
> > > > > >>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> > > > > >>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI
> > > became
> > > > > >>> cool :)
> > > > > >>>>>> Regarding docker-slave-02, I'll try to replace it after
> waiting
> > > for
> > > > > a
> > > > > >>>>>> while to make sure there's no objection.
> > > > > >>>>>>
> > > > > >>>>>> Kengo Seki <se...@apache.org>
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org>
> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks a lot for the update, Olaf!
> > > > > >>>>>>>
> > > > > >>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > > > > >>>>>>>
> > > > > >>>>>>>> Hi,
> > > > > >>>>>>>>
> > > > > >>>>>>>> All machines patched. Jenkins and it plugins are updated:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Things to be noted:
> > > > > >>>>>>>>
> > > > > >>>>>>>> * Slave 2 seems to be in serious problems. The disk image
> > > seems to
> > > > > >>> be
> > > > > >>>>>>>> corrupt, I would say:
> > > > > >>>>>>>> One of the problems: docker does not start any more.
> > > > > >>>>>>>> Is there anything important on it ? If yes please contact
> me.
> > > I
> > > > > >>> would
> > > > > >>>>>>>> recommend to set up slave2 from scratch again.
> > > > > >>>>>>>>
> > > > > >>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It
> now
> > > > > >>> imposes
> > > > > >>>>>>>> stricter rules. Not sure if there is a job depending on
> it.
> > > > > >>>>>>>>
> > > > > >>>>>>>> * I removed the CVS plugin.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Everything else seem to working as usual.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Best,
> > > > > >>>>>>>> Olaf
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <
> of@oflebbe.de>:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Hi,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I am doing an update of the machines in CI . Seems a
> couple
> > > of
> > > > > >>> security
> > > > > >>>>>>>> fixes are to be applied.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Olaf
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>
> > > > >
> > > > >
> > >
>

Re: Update

[Kengo Seki <se...@apache.org>]

I intended to keep current data in the storage and only shrink its
capacity, because I thought slaves also have some settings on the
disk.
But are following steps enough for slave setup? If so, I'll wipe and
setup them from scratch.

* launch EC2 instance
* create jenkins user
* configure its authorized_keys so that master can login with current key
* change slave's IP address on master
* (then master automatically configure this node, e.g., installing
agent, right?)

Kengo Seki <se...@apache.org>

On Tue, Nov 10, 2020 at 4:09 PM Evans Ye <ev...@apache.org> wrote:
>
> Yes I think overall your plan is good.
> What's the purpose of leveraging EBS snapshot? Is it to backup the things
> we have before migration?
> Except for the master node(have jenkins settings stored on disk), all those
> slaves can be wiped out directly.
>
>
>
> Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午2:42寫道：
>
> > Thanks everyone for the information! Now I understand our circumstances.
> > So we're going to split two 1TB volumes attached to slave06 and 07
> > into four 500GB volumes (and change their type to gp2), reattach them
> > to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> > right?
> >
> > > Kengo would you like to take this, or you need a help?
> >
> > I think I can do them somehow (maybe using EBS snapshot?), but let me
> > ask your help if I'm stuck. :)
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
> > >
> > > OK. I got it now.
> > > So the newly created volumes are currently attached to slave06_2 and
> > > slave07_2, respectively.
> > > However, they're standard HDD, not GP2 SSD. I think we can take this
> > chance
> > > to recreate those 2 slaves and do an overhaul of our infrastructure.
> > >
> > > Kengo would you like to take this, or you need a help?
> > >
> > > Evans
> > >
> > > Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
> > >
> > > > Hi,
> > > >
> > > > OMG . I think I did it.
> > > >
> > > > A few years ago two of the instance had a hardware problems and did not
> > > > reboot any more, filesystem was corrupted and so on.  That was at the
> > time
> > > > of the spectre vulnarability discovery. (2018) . At that time AWS had
> > major
> > > > instabilities since updating firmware seem to have failed for some
> > classes
> > > > of hardware.
> > > >
> > > > I tried to recreate them as close as possible but I may have left
> > > > accidentely the volumes around. Please lets delete them.
> > > >
> > > > Olaf
> > > >
> > > > > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
> > > > >
> > > > > Thanks Evans!
> > > > >
> > > > > It's great you found the details: they are definitely accurate as I
> > am
> > > > > recalling now. Kengo, do you think splitting the volumes would help
> > us
> > > > for a
> > > > > while? Or perhaps we shall try to expand the resource pool (which
> > might
> > > > take a
> > > > > while)?
> > > > >
> > > > > Thanks!
> > > > >  Cos
> > > > >
> > > > > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > > > >> In fact, the original deal of our resource is as follows:
> > > > >>
> > > > >>> 1 m3.2xlarge for CI
> > > > >>> 4 m3.xlarge for CI and demo
> > > > >>> 3 1TB EBS volumes
> > > > >>> 5 elastic IP addresses
> > > > >>
> > > > >> So technically we should not use that 2 additional 1T volumes
> > (created
> > > > in
> > > > >> 2018).
> > > > >> Instead, I think what we can do is to split up one of the existing
> > 1TB
> > > > >> volumes(ex: attached to slave07) into smaller volumes for slave02,
> > 03.
> > > > >>
> > > > >>
> > > > >> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> > > > >>
> > > > >>> Kengo,
> > > > >>>
> > > > >>> We had an agreement with EMR folks that we are using the resources
> > > > >>> available
> > > > >>> to us and it is included into their budget (or something to this
> > > > extent).
> > > > >>> If
> > > > >>> you see some of the resources available under our account - I
> > don't see
> > > > >>> why we
> > > > >>> can't use them.
> > > > >>>
> > > > >>> If for whatever reason we need to expand the pool, that would
> > require a
> > > > >>> separate conversation with nice folks from that team, I imagine.
> > Please
> > > > >>> let me
> > > > >>> know if I can help with this going forward.
> > > > >>>
> > > > >>> Thanks!
> > > > >>>  Cos
> > > > >>>
> > > > >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > > > >>>> Thanks for the comment, Cos! I was able to start docker service on
> > > > >>>> docker-slave-02 without replacing and am running some Jenkins
> > jobs on
> > > > >>>> it now, so I'll replace it in the short future.
> > > > >>>> I have a few things that I'd like to ask additionally:
> > > > >>>>
> > > > >>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that
> > has
> > > > >>>> only 8GiB capacity, and they sometimes run short and stop the CI.
> > > > >>>>  May I increase them to 20 or 30 GiB when I replace those
> > instances?
> > > > >>>> (I'm not sure what is our budget)
> > > > >>>>
> > > > >>>> * They use an instance store with 30GiB to put docker images into
> > it,
> > > > >>>> and they also sometimes run short.
> > > > >>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
> > > > >>>> vol-4efa69ae) on AWS console.
> > > > >>>>  May I attach them to 02 and 03 instead of instance stores, or are
> > > > >>>> they backups or something?
> > > > >>>>
> > > > >>>> Kengo Seki <se...@apache.org>
> > > > >>>>
> > > > >>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <cos@apache.org
> > >
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>> I'd say let replace the broken one. I don't think there's a
> > > > sentimental
> > > > >>>>> value attached ;)
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> With regards,
> > > > >>>>>   Cos
> > > > >>>>>
> > > > >>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> > > > >>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI
> > became
> > > > >>> cool :)
> > > > >>>>>> Regarding docker-slave-02, I'll try to replace it after waiting
> > for
> > > > a
> > > > >>>>>> while to make sure there's no objection.
> > > > >>>>>>
> > > > >>>>>> Kengo Seki <se...@apache.org>
> > > > >>>>>>
> > > > >>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Thanks a lot for the update, Olaf!
> > > > >>>>>>>
> > > > >>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > > > >>>>>>>
> > > > >>>>>>>> Hi,
> > > > >>>>>>>>
> > > > >>>>>>>> All machines patched. Jenkins and it plugins are updated:
> > > > >>>>>>>>
> > > > >>>>>>>> Things to be noted:
> > > > >>>>>>>>
> > > > >>>>>>>> * Slave 2 seems to be in serious problems. The disk image
> > seems to
> > > > >>> be
> > > > >>>>>>>> corrupt, I would say:
> > > > >>>>>>>> One of the problems: docker does not start any more.
> > > > >>>>>>>> Is there anything important on it ? If yes please contact me.
> > I
> > > > >>> would
> > > > >>>>>>>> recommend to set up slave2 from scratch again.
> > > > >>>>>>>>
> > > > >>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
> > > > >>> imposes
> > > > >>>>>>>> stricter rules. Not sure if there is a job depending on it.
> > > > >>>>>>>>
> > > > >>>>>>>> * I removed the CVS plugin.
> > > > >>>>>>>>
> > > > >>>>>>>> Everything else seem to working as usual.
> > > > >>>>>>>>
> > > > >>>>>>>> Best,
> > > > >>>>>>>> Olaf
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > > > >>>>>>>>>
> > > > >>>>>>>>> Hi,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I am doing an update of the machines in CI . Seems a couple
> > of
> > > > >>> security
> > > > >>>>>>>> fixes are to be applied.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Olaf
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>
> > > >
> > > >
> >

Re: Update

[Evans Ye <ev...@apache.org>]

Yes I think overall your plan is good.
What's the purpose of leveraging EBS snapshot? Is it to backup the things
we have before migration?
Except for the master node(have jenkins settings stored on disk), all those
slaves can be wiped out directly.



Kengo Seki <se...@apache.org> 於 2020年11月10日 週二 下午2:42寫道：

> Thanks everyone for the information! Now I understand our circumstances.
> So we're going to split two 1TB volumes attached to slave06 and 07
> into four 500GB volumes (and change their type to gp2), reattach them
> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> right?
>
> > Kengo would you like to take this, or you need a help?
>
> I think I can do them somehow (maybe using EBS snapshot?), but let me
> ask your help if I'm stuck. :)
>
> Kengo Seki <se...@apache.org>
>
> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
> >
> > OK. I got it now.
> > So the newly created volumes are currently attached to slave06_2 and
> > slave07_2, respectively.
> > However, they're standard HDD, not GP2 SSD. I think we can take this
> chance
> > to recreate those 2 slaves and do an overhaul of our infrastructure.
> >
> > Kengo would you like to take this, or you need a help?
> >
> > Evans
> >
> > Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
> >
> > > Hi,
> > >
> > > OMG . I think I did it.
> > >
> > > A few years ago two of the instance had a hardware problems and did not
> > > reboot any more, filesystem was corrupted and so on.  That was at the
> time
> > > of the spectre vulnarability discovery. (2018) . At that time AWS had
> major
> > > instabilities since updating firmware seem to have failed for some
> classes
> > > of hardware.
> > >
> > > I tried to recreate them as close as possible but I may have left
> > > accidentely the volumes around. Please lets delete them.
> > >
> > > Olaf
> > >
> > > > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
> > > >
> > > > Thanks Evans!
> > > >
> > > > It's great you found the details: they are definitely accurate as I
> am
> > > > recalling now. Kengo, do you think splitting the volumes would help
> us
> > > for a
> > > > while? Or perhaps we shall try to expand the resource pool (which
> might
> > > take a
> > > > while)?
> > > >
> > > > Thanks!
> > > >  Cos
> > > >
> > > > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > > >> In fact, the original deal of our resource is as follows:
> > > >>
> > > >>> 1 m3.2xlarge for CI
> > > >>> 4 m3.xlarge for CI and demo
> > > >>> 3 1TB EBS volumes
> > > >>> 5 elastic IP addresses
> > > >>
> > > >> So technically we should not use that 2 additional 1T volumes
> (created
> > > in
> > > >> 2018).
> > > >> Instead, I think what we can do is to split up one of the existing
> 1TB
> > > >> volumes(ex: attached to slave07) into smaller volumes for slave02,
> 03.
> > > >>
> > > >>
> > > >> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> > > >>
> > > >>> Kengo,
> > > >>>
> > > >>> We had an agreement with EMR folks that we are using the resources
> > > >>> available
> > > >>> to us and it is included into their budget (or something to this
> > > extent).
> > > >>> If
> > > >>> you see some of the resources available under our account - I
> don't see
> > > >>> why we
> > > >>> can't use them.
> > > >>>
> > > >>> If for whatever reason we need to expand the pool, that would
> require a
> > > >>> separate conversation with nice folks from that team, I imagine.
> Please
> > > >>> let me
> > > >>> know if I can help with this going forward.
> > > >>>
> > > >>> Thanks!
> > > >>>  Cos
> > > >>>
> > > >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > > >>>> Thanks for the comment, Cos! I was able to start docker service on
> > > >>>> docker-slave-02 without replacing and am running some Jenkins
> jobs on
> > > >>>> it now, so I'll replace it in the short future.
> > > >>>> I have a few things that I'd like to ask additionally:
> > > >>>>
> > > >>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that
> has
> > > >>>> only 8GiB capacity, and they sometimes run short and stop the CI.
> > > >>>>  May I increase them to 20 or 30 GiB when I replace those
> instances?
> > > >>>> (I'm not sure what is our budget)
> > > >>>>
> > > >>>> * They use an instance store with 30GiB to put docker images into
> it,
> > > >>>> and they also sometimes run short.
> > > >>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
> > > >>>> vol-4efa69ae) on AWS console.
> > > >>>>  May I attach them to 02 and 03 instead of instance stores, or are
> > > >>>> they backups or something?
> > > >>>>
> > > >>>> Kengo Seki <se...@apache.org>
> > > >>>>
> > > >>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <cos@apache.org
> >
> > > >>> wrote:
> > > >>>>>
> > > >>>>> I'd say let replace the broken one. I don't think there's a
> > > sentimental
> > > >>>>> value attached ;)
> > > >>>>>
> > > >>>>> --
> > > >>>>> With regards,
> > > >>>>>   Cos
> > > >>>>>
> > > >>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> > > >>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI
> became
> > > >>> cool :)
> > > >>>>>> Regarding docker-slave-02, I'll try to replace it after waiting
> for
> > > a
> > > >>>>>> while to make sure there's no objection.
> > > >>>>>>
> > > >>>>>> Kengo Seki <se...@apache.org>
> > > >>>>>>
> > > >>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> > > >>>>>>>
> > > >>>>>>> Thanks a lot for the update, Olaf!
> > > >>>>>>>
> > > >>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > > >>>>>>>
> > > >>>>>>>> Hi,
> > > >>>>>>>>
> > > >>>>>>>> All machines patched. Jenkins and it plugins are updated:
> > > >>>>>>>>
> > > >>>>>>>> Things to be noted:
> > > >>>>>>>>
> > > >>>>>>>> * Slave 2 seems to be in serious problems. The disk image
> seems to
> > > >>> be
> > > >>>>>>>> corrupt, I would say:
> > > >>>>>>>> One of the problems: docker does not start any more.
> > > >>>>>>>> Is there anything important on it ? If yes please contact me.
> I
> > > >>> would
> > > >>>>>>>> recommend to set up slave2 from scratch again.
> > > >>>>>>>>
> > > >>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
> > > >>> imposes
> > > >>>>>>>> stricter rules. Not sure if there is a job depending on it.
> > > >>>>>>>>
> > > >>>>>>>> * I removed the CVS plugin.
> > > >>>>>>>>
> > > >>>>>>>> Everything else seem to working as usual.
> > > >>>>>>>>
> > > >>>>>>>> Best,
> > > >>>>>>>> Olaf
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > > >>>>>>>>>
> > > >>>>>>>>> Hi,
> > > >>>>>>>>>
> > > >>>>>>>>> I am doing an update of the machines in CI . Seems a couple
> of
> > > >>> security
> > > >>>>>>>> fixes are to be applied.
> > > >>>>>>>>>
> > > >>>>>>>>> Olaf
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>
> > >
> > >
>

Re: Update

[Kengo Seki <se...@apache.org>]

Thanks everyone for the information! Now I understand our circumstances.
So we're going to split two 1TB volumes attached to slave06 and 07
into four 500GB volumes (and change their type to gp2), reattach them
to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
right?

> Kengo would you like to take this, or you need a help?

I think I can do them somehow (maybe using EBS snapshot?), but let me
ask your help if I'm stuck. :)

Kengo Seki <se...@apache.org>

On Tue, Nov 10, 2020 at 1:00 AM Evans Ye <ev...@apache.org> wrote:
>
> OK. I got it now.
> So the newly created volumes are currently attached to slave06_2 and
> slave07_2, respectively.
> However, they're standard HDD, not GP2 SSD. I think we can take this chance
> to recreate those 2 slaves and do an overhaul of our infrastructure.
>
> Kengo would you like to take this, or you need a help?
>
> Evans
>
> Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：
>
> > Hi,
> >
> > OMG . I think I did it.
> >
> > A few years ago two of the instance had a hardware problems and did not
> > reboot any more, filesystem was corrupted and so on.  That was at the time
> > of the spectre vulnarability discovery. (2018) . At that time AWS had major
> > instabilities since updating firmware seem to have failed for some classes
> > of hardware.
> >
> > I tried to recreate them as close as possible but I may have left
> > accidentely the volumes around. Please lets delete them.
> >
> > Olaf
> >
> > > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
> > >
> > > Thanks Evans!
> > >
> > > It's great you found the details: they are definitely accurate as I am
> > > recalling now. Kengo, do you think splitting the volumes would help us
> > for a
> > > while? Or perhaps we shall try to expand the resource pool (which might
> > take a
> > > while)?
> > >
> > > Thanks!
> > >  Cos
> > >
> > > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > >> In fact, the original deal of our resource is as follows:
> > >>
> > >>> 1 m3.2xlarge for CI
> > >>> 4 m3.xlarge for CI and demo
> > >>> 3 1TB EBS volumes
> > >>> 5 elastic IP addresses
> > >>
> > >> So technically we should not use that 2 additional 1T volumes (created
> > in
> > >> 2018).
> > >> Instead, I think what we can do is to split up one of the existing 1TB
> > >> volumes(ex: attached to slave07) into smaller volumes for slave02, 03.
> > >>
> > >>
> > >> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> > >>
> > >>> Kengo,
> > >>>
> > >>> We had an agreement with EMR folks that we are using the resources
> > >>> available
> > >>> to us and it is included into their budget (or something to this
> > extent).
> > >>> If
> > >>> you see some of the resources available under our account - I don't see
> > >>> why we
> > >>> can't use them.
> > >>>
> > >>> If for whatever reason we need to expand the pool, that would require a
> > >>> separate conversation with nice folks from that team, I imagine. Please
> > >>> let me
> > >>> know if I can help with this going forward.
> > >>>
> > >>> Thanks!
> > >>>  Cos
> > >>>
> > >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > >>>> Thanks for the comment, Cos! I was able to start docker service on
> > >>>> docker-slave-02 without replacing and am running some Jenkins jobs on
> > >>>> it now, so I'll replace it in the short future.
> > >>>> I have a few things that I'd like to ask additionally:
> > >>>>
> > >>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> > >>>> only 8GiB capacity, and they sometimes run short and stop the CI.
> > >>>>  May I increase them to 20 or 30 GiB when I replace those instances?
> > >>>> (I'm not sure what is our budget)
> > >>>>
> > >>>> * They use an instance store with 30GiB to put docker images into it,
> > >>>> and they also sometimes run short.
> > >>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
> > >>>> vol-4efa69ae) on AWS console.
> > >>>>  May I attach them to 02 and 03 instead of instance stores, or are
> > >>>> they backups or something?
> > >>>>
> > >>>> Kengo Seki <se...@apache.org>
> > >>>>
> > >>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org>
> > >>> wrote:
> > >>>>>
> > >>>>> I'd say let replace the broken one. I don't think there's a
> > sentimental
> > >>>>> value attached ;)
> > >>>>>
> > >>>>> --
> > >>>>> With regards,
> > >>>>>   Cos
> > >>>>>
> > >>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> > >>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI became
> > >>> cool :)
> > >>>>>> Regarding docker-slave-02, I'll try to replace it after waiting for
> > a
> > >>>>>> while to make sure there's no objection.
> > >>>>>>
> > >>>>>> Kengo Seki <se...@apache.org>
> > >>>>>>
> > >>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> > >>>>>>>
> > >>>>>>> Thanks a lot for the update, Olaf!
> > >>>>>>>
> > >>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > >>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> All machines patched. Jenkins and it plugins are updated:
> > >>>>>>>>
> > >>>>>>>> Things to be noted:
> > >>>>>>>>
> > >>>>>>>> * Slave 2 seems to be in serious problems. The disk image seems to
> > >>> be
> > >>>>>>>> corrupt, I would say:
> > >>>>>>>> One of the problems: docker does not start any more.
> > >>>>>>>> Is there anything important on it ? If yes please contact me. I
> > >>> would
> > >>>>>>>> recommend to set up slave2 from scratch again.
> > >>>>>>>>
> > >>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
> > >>> imposes
> > >>>>>>>> stricter rules. Not sure if there is a job depending on it.
> > >>>>>>>>
> > >>>>>>>> * I removed the CVS plugin.
> > >>>>>>>>
> > >>>>>>>> Everything else seem to working as usual.
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Olaf
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > >>>>>>>>>
> > >>>>>>>>> Hi,
> > >>>>>>>>>
> > >>>>>>>>> I am doing an update of the machines in CI . Seems a couple of
> > >>> security
> > >>>>>>>> fixes are to be applied.
> > >>>>>>>>>
> > >>>>>>>>> Olaf
> > >>>>>>>>
> > >>>>>>>>
> > >>>
> >
> >

Re: Update

[Evans Ye <ev...@apache.org>]

OK. I got it now.
So the newly created volumes are currently attached to slave06_2 and
slave07_2, respectively.
However, they're standard HDD, not GP2 SSD. I think we can take this chance
to recreate those 2 slaves and do an overhaul of our infrastructure.

Kengo would you like to take this, or you need a help?

Evans

Olaf Flebbe <of...@oflebbe.de> 於 2020年11月6日 週五 上午2:40寫道：

> Hi,
>
> OMG . I think I did it.
>
> A few years ago two of the instance had a hardware problems and did not
> reboot any more, filesystem was corrupted and so on.  That was at the time
> of the spectre vulnarability discovery. (2018) . At that time AWS had major
> instabilities since updating firmware seem to have failed for some classes
> of hardware.
>
> I tried to recreate them as close as possible but I may have left
> accidentely the volumes around. Please lets delete them.
>
> Olaf
>
> > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
> >
> > Thanks Evans!
> >
> > It's great you found the details: they are definitely accurate as I am
> > recalling now. Kengo, do you think splitting the volumes would help us
> for a
> > while? Or perhaps we shall try to expand the resource pool (which might
> take a
> > while)?
> >
> > Thanks!
> >  Cos
> >
> > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> >> In fact, the original deal of our resource is as follows:
> >>
> >>> 1 m3.2xlarge for CI
> >>> 4 m3.xlarge for CI and demo
> >>> 3 1TB EBS volumes
> >>> 5 elastic IP addresses
> >>
> >> So technically we should not use that 2 additional 1T volumes (created
> in
> >> 2018).
> >> Instead, I think what we can do is to split up one of the existing 1TB
> >> volumes(ex: attached to slave07) into smaller volumes for slave02, 03.
> >>
> >>
> >> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> >>
> >>> Kengo,
> >>>
> >>> We had an agreement with EMR folks that we are using the resources
> >>> available
> >>> to us and it is included into their budget (or something to this
> extent).
> >>> If
> >>> you see some of the resources available under our account - I don't see
> >>> why we
> >>> can't use them.
> >>>
> >>> If for whatever reason we need to expand the pool, that would require a
> >>> separate conversation with nice folks from that team, I imagine. Please
> >>> let me
> >>> know if I can help with this going forward.
> >>>
> >>> Thanks!
> >>>  Cos
> >>>
> >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> >>>> Thanks for the comment, Cos! I was able to start docker service on
> >>>> docker-slave-02 without replacing and am running some Jenkins jobs on
> >>>> it now, so I'll replace it in the short future.
> >>>> I have a few things that I'd like to ask additionally:
> >>>>
> >>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> >>>> only 8GiB capacity, and they sometimes run short and stop the CI.
> >>>>  May I increase them to 20 or 30 GiB when I replace those instances?
> >>>> (I'm not sure what is our budget)
> >>>>
> >>>> * They use an instance store with 30GiB to put docker images into it,
> >>>> and they also sometimes run short.
> >>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
> >>>> vol-4efa69ae) on AWS console.
> >>>>  May I attach them to 02 and 03 instead of instance stores, or are
> >>>> they backups or something?
> >>>>
> >>>> Kengo Seki <se...@apache.org>
> >>>>
> >>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org>
> >>> wrote:
> >>>>>
> >>>>> I'd say let replace the broken one. I don't think there's a
> sentimental
> >>>>> value attached ;)
> >>>>>
> >>>>> --
> >>>>> With regards,
> >>>>>   Cos
> >>>>>
> >>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> >>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI became
> >>> cool :)
> >>>>>> Regarding docker-slave-02, I'll try to replace it after waiting for
> a
> >>>>>> while to make sure there's no objection.
> >>>>>>
> >>>>>> Kengo Seki <se...@apache.org>
> >>>>>>
> >>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> >>>>>>>
> >>>>>>> Thanks a lot for the update, Olaf!
> >>>>>>>
> >>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> All machines patched. Jenkins and it plugins are updated:
> >>>>>>>>
> >>>>>>>> Things to be noted:
> >>>>>>>>
> >>>>>>>> * Slave 2 seems to be in serious problems. The disk image seems to
> >>> be
> >>>>>>>> corrupt, I would say:
> >>>>>>>> One of the problems: docker does not start any more.
> >>>>>>>> Is there anything important on it ? If yes please contact me. I
> >>> would
> >>>>>>>> recommend to set up slave2 from scratch again.
> >>>>>>>>
> >>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
> >>> imposes
> >>>>>>>> stricter rules. Not sure if there is a job depending on it.
> >>>>>>>>
> >>>>>>>> * I removed the CVS plugin.
> >>>>>>>>
> >>>>>>>> Everything else seem to working as usual.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Olaf
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> >>>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I am doing an update of the machines in CI . Seems a couple of
> >>> security
> >>>>>>>> fixes are to be applied.
> >>>>>>>>>
> >>>>>>>>> Olaf
> >>>>>>>>
> >>>>>>>>
> >>>
>
>

Re: Update

[Olaf Flebbe <of...@oflebbe.de>]

Hi,

OMG . I think I did it.

A few years ago two of the instance had a hardware problems and did not reboot any more, filesystem was corrupted and so on.  That was at the time of the spectre vulnarability discovery. (2018) . At that time AWS had major instabilities since updating firmware seem to have failed for some classes of hardware.

I tried to recreate them as close as possible but I may have left accidentely the volumes around. Please lets delete them.

Olaf

> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <co...@apache.org>:
> 
> Thanks Evans!
> 
> It's great you found the details: they are definitely accurate as I am
> recalling now. Kengo, do you think splitting the volumes would help us for a
> while? Or perhaps we shall try to expand the resource pool (which might take a
> while)? 
> 
> Thanks!
>  Cos
> 
> On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
>> In fact, the original deal of our resource is as follows:
>> 
>>> 1 m3.2xlarge for CI
>>> 4 m3.xlarge for CI and demo
>>> 3 1TB EBS volumes
>>> 5 elastic IP addresses
>> 
>> So technically we should not use that 2 additional 1T volumes (created in
>> 2018).
>> Instead, I think what we can do is to split up one of the existing 1TB
>> volumes(ex: attached to slave07) into smaller volumes for slave02, 03.
>> 
>> 
>> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
>> 
>>> Kengo,
>>> 
>>> We had an agreement with EMR folks that we are using the resources
>>> available
>>> to us and it is included into their budget (or something to this extent).
>>> If
>>> you see some of the resources available under our account - I don't see
>>> why we
>>> can't use them.
>>> 
>>> If for whatever reason we need to expand the pool, that would require a
>>> separate conversation with nice folks from that team, I imagine. Please
>>> let me
>>> know if I can help with this going forward.
>>> 
>>> Thanks!
>>>  Cos
>>> 
>>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
>>>> Thanks for the comment, Cos! I was able to start docker service on
>>>> docker-slave-02 without replacing and am running some Jenkins jobs on
>>>> it now, so I'll replace it in the short future.
>>>> I have a few things that I'd like to ask additionally:
>>>> 
>>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that has
>>>> only 8GiB capacity, and they sometimes run short and stop the CI.
>>>>  May I increase them to 20 or 30 GiB when I replace those instances?
>>>> (I'm not sure what is our budget)
>>>> 
>>>> * They use an instance store with 30GiB to put docker images into it,
>>>> and they also sometimes run short.
>>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
>>>> vol-4efa69ae) on AWS console.
>>>>  May I attach them to 02 and 03 instead of instance stores, or are
>>>> they backups or something?
>>>> 
>>>> Kengo Seki <se...@apache.org>
>>>> 
>>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org>
>>> wrote:
>>>>> 
>>>>> I'd say let replace the broken one. I don't think there's a sentimental
>>>>> value attached ;)
>>>>> 
>>>>> --
>>>>> With regards,
>>>>>   Cos
>>>>> 
>>>>> On 02.11.2020 08:16, Kengo Seki wrote:
>>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI became
>>> cool :)
>>>>>> Regarding docker-slave-02, I'll try to replace it after waiting for a
>>>>>> while to make sure there's no objection.
>>>>>> 
>>>>>> Kengo Seki <se...@apache.org>
>>>>>> 
>>>>>> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
>>>>>>> 
>>>>>>> Thanks a lot for the update, Olaf!
>>>>>>> 
>>>>>>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> All machines patched. Jenkins and it plugins are updated:
>>>>>>>> 
>>>>>>>> Things to be noted:
>>>>>>>> 
>>>>>>>> * Slave 2 seems to be in serious problems. The disk image seems to
>>> be
>>>>>>>> corrupt, I would say:
>>>>>>>> One of the problems: docker does not start any more.
>>>>>>>> Is there anything important on it ? If yes please contact me. I
>>> would
>>>>>>>> recommend to set up slave2 from scratch again.
>>>>>>>> 
>>>>>>>> * There was a warning regarding Copy Artifacts Plugin. It now
>>> imposes
>>>>>>>> stricter rules. Not sure if there is a job depending on it.
>>>>>>>> 
>>>>>>>> * I removed the CVS plugin.
>>>>>>>> 
>>>>>>>> Everything else seem to working as usual.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Olaf
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I am doing an update of the machines in CI . Seems a couple of
>>> security
>>>>>>>> fixes are to be applied.
>>>>>>>>> 
>>>>>>>>> Olaf
>>>>>>>> 
>>>>>>>> 
>>> 

Re: Update

[Konstantin Boudnik <co...@apache.org>]

Thanks Evans!

It's great you found the details: they are definitely accurate as I am
recalling now. Kengo, do you think splitting the volumes would help us for a
while? Or perhaps we shall try to expand the resource pool (which might take a
while)? 

Thanks!
  Cos

On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> In fact, the original deal of our resource is as follows:
> 
> > 1 m3.2xlarge for CI
> > 4 m3.xlarge for CI and demo
> > 3 1TB EBS volumes
> > 5 elastic IP addresses
> 
> So technically we should not use that 2 additional 1T volumes (created in
> 2018).
> Instead, I think what we can do is to split up one of the existing 1TB
> volumes(ex: attached to slave07) into smaller volumes for slave02, 03.
> 
> 
> Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：
> 
> > Kengo,
> >
> > We had an agreement with EMR folks that we are using the resources
> > available
> > to us and it is included into their budget (or something to this extent).
> > If
> > you see some of the resources available under our account - I don't see
> > why we
> > can't use them.
> >
> > If for whatever reason we need to expand the pool, that would require a
> > separate conversation with nice folks from that team, I imagine. Please
> > let me
> > know if I can help with this going forward.
> >
> > Thanks!
> >   Cos
> >
> > On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > > Thanks for the comment, Cos! I was able to start docker service on
> > > docker-slave-02 without replacing and am running some Jenkins jobs on
> > > it now, so I'll replace it in the short future.
> > > I have a few things that I'd like to ask additionally:
> > >
> > > * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> > > only 8GiB capacity, and they sometimes run short and stop the CI.
> > >   May I increase them to 20 or 30 GiB when I replace those instances?
> > > (I'm not sure what is our budget)
> > >
> > > * They use an instance store with 30GiB to put docker images into it,
> > > and they also sometimes run short.
> > >   It seems there are two unused volumes with 1TiB (vol-ae71114e and
> > > vol-4efa69ae) on AWS console.
> > >   May I attach them to 02 and 03 instead of instance stores, or are
> > > they backups or something?
> > >
> > > Kengo Seki <se...@apache.org>
> > >
> > > On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > > I'd say let replace the broken one. I don't think there's a sentimental
> > > > value attached ;)
> > > >
> > > > --
> > > > With regards,
> > > >    Cos
> > > >
> > > > On 02.11.2020 08:16, Kengo Seki wrote:
> > > > > Thanks for updating Olaf! I've just noticed the Jenkins UI became
> > cool :)
> > > > > Regarding docker-slave-02, I'll try to replace it after waiting for a
> > > > > while to make sure there's no objection.
> > > > >
> > > > > Kengo Seki <se...@apache.org>
> > > > >
> > > > > On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> > > > >>
> > > > >> Thanks a lot for the update, Olaf!
> > > > >>
> > > > >> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > > > >>
> > > > >>> Hi,
> > > > >>>
> > > > >>> All machines patched. Jenkins and it plugins are updated:
> > > > >>>
> > > > >>> Things to be noted:
> > > > >>>
> > > > >>> * Slave 2 seems to be in serious problems. The disk image seems to
> > be
> > > > >>> corrupt, I would say:
> > > > >>> One of the problems: docker does not start any more.
> > > > >>> Is there anything important on it ? If yes please contact me. I
> > would
> > > > >>> recommend to set up slave2 from scratch again.
> > > > >>>
> > > > >>> * There was a warning regarding Copy Artifacts Plugin. It now
> > imposes
> > > > >>> stricter rules. Not sure if there is a job depending on it.
> > > > >>>
> > > > >>> * I removed the CVS plugin.
> > > > >>>
> > > > >>> Everything else seem to working as usual.
> > > > >>>
> > > > >>> Best,
> > > > >>> Olaf
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > > > >>>>
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> I am doing an update of the machines in CI . Seems a couple of
> > security
> > > > >>> fixes are to be applied.
> > > > >>>>
> > > > >>>> Olaf
> > > > >>>
> > > > >>>
> >

Re: Update

[Evans Ye <ev...@apache.org>]

In fact, the original deal of our resource is as follows:

> 1 m3.2xlarge for CI
> 4 m3.xlarge for CI and demo
> 3 1TB EBS volumes
> 5 elastic IP addresses

So technically we should not use that 2 additional 1T volumes (created in
2018).
Instead, I think what we can do is to split up one of the existing 1TB
volumes(ex: attached to slave07) into smaller volumes for slave02, 03.


Konstantin Boudnik <co...@apache.org> 於 2020年11月4日 週三 下午2:28寫道：

> Kengo,
>
> We had an agreement with EMR folks that we are using the resources
> available
> to us and it is included into their budget (or something to this extent).
> If
> you see some of the resources available under our account - I don't see
> why we
> can't use them.
>
> If for whatever reason we need to expand the pool, that would require a
> separate conversation with nice folks from that team, I imagine. Please
> let me
> know if I can help with this going forward.
>
> Thanks!
>   Cos
>
> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > Thanks for the comment, Cos! I was able to start docker service on
> > docker-slave-02 without replacing and am running some Jenkins jobs on
> > it now, so I'll replace it in the short future.
> > I have a few things that I'd like to ask additionally:
> >
> > * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> > only 8GiB capacity, and they sometimes run short and stop the CI.
> >   May I increase them to 20 or 30 GiB when I replace those instances?
> > (I'm not sure what is our budget)
> >
> > * They use an instance store with 30GiB to put docker images into it,
> > and they also sometimes run short.
> >   It seems there are two unused volumes with 1TiB (vol-ae71114e and
> > vol-4efa69ae) on AWS console.
> >   May I attach them to 02 and 03 instead of instance stores, or are
> > they backups or something?
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org>
> wrote:
> > >
> > > I'd say let replace the broken one. I don't think there's a sentimental
> > > value attached ;)
> > >
> > > --
> > > With regards,
> > >    Cos
> > >
> > > On 02.11.2020 08:16, Kengo Seki wrote:
> > > > Thanks for updating Olaf! I've just noticed the Jenkins UI became
> cool :)
> > > > Regarding docker-slave-02, I'll try to replace it after waiting for a
> > > > while to make sure there's no objection.
> > > >
> > > > Kengo Seki <se...@apache.org>
> > > >
> > > > On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> > > >>
> > > >> Thanks a lot for the update, Olaf!
> > > >>
> > > >> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> All machines patched. Jenkins and it plugins are updated:
> > > >>>
> > > >>> Things to be noted:
> > > >>>
> > > >>> * Slave 2 seems to be in serious problems. The disk image seems to
> be
> > > >>> corrupt, I would say:
> > > >>> One of the problems: docker does not start any more.
> > > >>> Is there anything important on it ? If yes please contact me. I
> would
> > > >>> recommend to set up slave2 from scratch again.
> > > >>>
> > > >>> * There was a warning regarding Copy Artifacts Plugin. It now
> imposes
> > > >>> stricter rules. Not sure if there is a job depending on it.
> > > >>>
> > > >>> * I removed the CVS plugin.
> > > >>>
> > > >>> Everything else seem to working as usual.
> > > >>>
> > > >>> Best,
> > > >>> Olaf
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > > >>>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> I am doing an update of the machines in CI . Seems a couple of
> security
> > > >>> fixes are to be applied.
> > > >>>>
> > > >>>> Olaf
> > > >>>
> > > >>>
>

Re: Update

[Konstantin Boudnik <co...@apache.org>]

Kengo,

We had an agreement with EMR folks that we are using the resources available
to us and it is included into their budget (or something to this extent). If
you see some of the resources available under our account - I don't see why we
can't use them.

If for whatever reason we need to expand the pool, that would require a
separate conversation with nice folks from that team, I imagine. Please let me
know if I can help with this going forward.

Thanks!
  Cos

On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> Thanks for the comment, Cos! I was able to start docker service on
> docker-slave-02 without replacing and am running some Jenkins jobs on
> it now, so I'll replace it in the short future.
> I have a few things that I'd like to ask additionally:
> 
> * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> only 8GiB capacity, and they sometimes run short and stop the CI.
>   May I increase them to 20 or 30 GiB when I replace those instances?
> (I'm not sure what is our budget)
> 
> * They use an instance store with 30GiB to put docker images into it,
> and they also sometimes run short.
>   It seems there are two unused volumes with 1TiB (vol-ae71114e and
> vol-4efa69ae) on AWS console.
>   May I attach them to 02 and 03 instead of instance stores, or are
> they backups or something?
> 
> Kengo Seki <se...@apache.org>
> 
> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org> wrote:
> >
> > I'd say let replace the broken one. I don't think there's a sentimental
> > value attached ;)
> >
> > --
> > With regards,
> >    Cos
> >
> > On 02.11.2020 08:16, Kengo Seki wrote:
> > > Thanks for updating Olaf! I've just noticed the Jenkins UI became cool :)
> > > Regarding docker-slave-02, I'll try to replace it after waiting for a
> > > while to make sure there's no objection.
> > >
> > > Kengo Seki <se...@apache.org>
> > >
> > > On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> > >>
> > >> Thanks a lot for the update, Olaf!
> > >>
> > >> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> > >>
> > >>> Hi,
> > >>>
> > >>> All machines patched. Jenkins and it plugins are updated:
> > >>>
> > >>> Things to be noted:
> > >>>
> > >>> * Slave 2 seems to be in serious problems. The disk image seems to be
> > >>> corrupt, I would say:
> > >>> One of the problems: docker does not start any more.
> > >>> Is there anything important on it ? If yes please contact me. I would
> > >>> recommend to set up slave2 from scratch again.
> > >>>
> > >>> * There was a warning regarding Copy Artifacts Plugin. It now imposes
> > >>> stricter rules. Not sure if there is a job depending on it.
> > >>>
> > >>> * I removed the CVS plugin.
> > >>>
> > >>> Everything else seem to working as usual.
> > >>>
> > >>> Best,
> > >>> Olaf
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I am doing an update of the machines in CI . Seems a couple of security
> > >>> fixes are to be applied.
> > >>>>
> > >>>> Olaf
> > >>>
> > >>>

Re: Update

[Kengo Seki <se...@apache.org>]

Thanks for the comment, Cos! I was able to start docker service on
docker-slave-02 without replacing and am running some Jenkins jobs on
it now, so I'll replace it in the short future.
I have a few things that I'd like to ask additionally:

* docker-slave-02 and 03 have a gp2 storage as a root volume that has
only 8GiB capacity, and they sometimes run short and stop the CI.
  May I increase them to 20 or 30 GiB when I replace those instances?
(I'm not sure what is our budget)

* They use an instance store with 30GiB to put docker images into it,
and they also sometimes run short.
  It seems there are two unused volumes with 1TiB (vol-ae71114e and
vol-4efa69ae) on AWS console.
  May I attach them to 02 and 03 instead of instance stores, or are
they backups or something?

Kengo Seki <se...@apache.org>

On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik <co...@apache.org> wrote:
>
> I'd say let replace the broken one. I don't think there's a sentimental
> value attached ;)
>
> --
> With regards,
>    Cos
>
> On 02.11.2020 08:16, Kengo Seki wrote:
> > Thanks for updating Olaf! I've just noticed the Jenkins UI became cool :)
> > Regarding docker-slave-02, I'll try to replace it after waiting for a
> > while to make sure there's no objection.
> >
> > Kengo Seki <se...@apache.org>
> >
> > On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
> >>
> >> Thanks a lot for the update, Olaf!
> >>
> >> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
> >>
> >>> Hi,
> >>>
> >>> All machines patched. Jenkins and it plugins are updated:
> >>>
> >>> Things to be noted:
> >>>
> >>> * Slave 2 seems to be in serious problems. The disk image seems to be
> >>> corrupt, I would say:
> >>> One of the problems: docker does not start any more.
> >>> Is there anything important on it ? If yes please contact me. I would
> >>> recommend to set up slave2 from scratch again.
> >>>
> >>> * There was a warning regarding Copy Artifacts Plugin. It now imposes
> >>> stricter rules. Not sure if there is a job depending on it.
> >>>
> >>> * I removed the CVS plugin.
> >>>
> >>> Everything else seem to working as usual.
> >>>
> >>> Best,
> >>> Olaf
> >>>
> >>>
> >>>
> >>>
> >>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I am doing an update of the machines in CI . Seems a couple of security
> >>> fixes are to be applied.
> >>>>
> >>>> Olaf
> >>>
> >>>

Re: Update

[Konstantin Boudnik <co...@apache.org>]

I'd say let replace the broken one. I don't think there's a sentimental 
value attached ;)

--
With regards,
   Cos

On 02.11.2020 08:16, Kengo Seki wrote:
> Thanks for updating Olaf! I've just noticed the Jenkins UI became cool :)
> Regarding docker-slave-02, I'll try to replace it after waiting for a
> while to make sure there's no objection.
> 
> Kengo Seki <se...@apache.org>
> 
> On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
>>
>> Thanks a lot for the update, Olaf!
>>
>> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
>>
>>> Hi,
>>>
>>> All machines patched. Jenkins and it plugins are updated:
>>>
>>> Things to be noted:
>>>
>>> * Slave 2 seems to be in serious problems. The disk image seems to be
>>> corrupt, I would say:
>>> One of the problems: docker does not start any more.
>>> Is there anything important on it ? If yes please contact me. I would
>>> recommend to set up slave2 from scratch again.
>>>
>>> * There was a warning regarding Copy Artifacts Plugin. It now imposes
>>> stricter rules. Not sure if there is a job depending on it.
>>>
>>> * I removed the CVS plugin.
>>>
>>> Everything else seem to working as usual.
>>>
>>> Best,
>>> Olaf
>>>
>>>
>>>
>>>
>>>> Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
>>>>
>>>> Hi,
>>>>
>>>> I am doing an update of the machines in CI . Seems a couple of security
>>> fixes are to be applied.
>>>>
>>>> Olaf
>>>
>>>

Re: Update

[Kengo Seki <se...@apache.org>]

Thanks for updating Olaf! I've just noticed the Jenkins UI became cool :)
Regarding docker-slave-02, I'll try to replace it after waiting for a
while to make sure there's no objection.

Kengo Seki <se...@apache.org>

On Mon, Nov 2, 2020 at 1:39 PM Jun HE <ju...@apache.org> wrote:
>
> Thanks a lot for the update, Olaf!
>
> Olaf Flebbe <of...@oflebbe.de> 于2020年10月31日周六 上午3:24写道：
>
> > Hi,
> >
> > All machines patched. Jenkins and it plugins are updated:
> >
> > Things to be noted:
> >
> > * Slave 2 seems to be in serious problems. The disk image seems to be
> > corrupt, I would say:
> > One of the problems: docker does not start any more.
> > Is there anything important on it ? If yes please contact me. I would
> > recommend to set up slave2 from scratch again.
> >
> > * There was a warning regarding Copy Artifacts Plugin. It now imposes
> > stricter rules. Not sure if there is a job depending on it.
> >
> > * I removed the CVS plugin.
> >
> > Everything else seem to working as usual.
> >
> > Best,
> > Olaf
> >
> >
> >
> >
> > > Am 30.10.2020 um 19:09 schrieb Olaf Flebbe <of...@oflebbe.de>:
> > >
> > > Hi,
> > >
> > > I am doing an update of the machines in CI . Seems a couple of security
> > fixes are to be applied.
> > >
> > > Olaf
> >
> >