You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Julien Eid <ju...@cloudera.com> on 2014/07/11 19:24:09 UTC
Jenkins builds are frozen
http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
Fedora 18 box is blocking jobs from running for the past two days.
http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
Stuck on archiving Sqoop artifacts.
Re: Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
The jenkins process actually ran out of the PermGen - fixed now. Thanks
On Wed, Jul 16, 2014 at 04:17PM, pengwenwu wrote:
> is possible network overloading cause failure as https://issues.jenkins-ci.org/browse/JENKINS-14332 ?
>
>
>
>
> Regards,
> Wenwu,Peng
>
>
>
>
>
>
>
> At 2014-07-16 02:49:26, "Konstantin Boudnik" <co...@apache.org> wrote:
> >Damn, centos6 slave still behaves - I think I will dive into this tomorrow to
> >find out why the stuff like this
> >
> >http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Datafu/label=centos6/513/console
> >
> >is happening. I have already rebooted it - perhaps the timing was wrong and
> >all these jobs failed because the slave was down? It is an unlikely
> >explanation, but I can't come up with anything better atm ;(
> >
> >Cos
> >
> >On Mon, Jul 14, 2014 at 10:50PM, Konstantin Boudnik wrote:
> >> Ok, Jenkins is at 1.572 now - let's see how the build proceeds from here.
> >>
> >> Cos
> >>
> >> On Mon, Jul 14, 2014 at 03:53PM, Konstantin Boudnik wrote:
> >> > Damn, I can't: only my home computer's keys are added to the CI setup ;( Ok, I
> >> > will do the upgrade in a couple of hours.
> >> >
> >> > On Mon, Jul 14, 2014 at 03:30PM, Konstantin Boudnik wrote:
> >> > > Good one Chandrasekhar! Ok, Jenkins is completely stuck right now, so let me
> >> > > start the upgrade right away and restart the server. We'll see how it goes
> >> > > after.
> >> > >
> >> > > Cos
> >> > >
> >> > > On Mon, Jul 14, 2014 at 02:13PM, Chandrasekhar Gopal wrote:
> >> > > > My 2 cents on Julien's comments about the OOM JIRAs. We had the same
> >> > > > issue when we installed version 1.560.
> >> > > >
> >> > > > <> Copying artifacts from other build jobs would hang
> >> > > > <> Slaves would disconnect at random.
> >> > > >
> >> > > > *However, as pointed out
> >> > > > in https://issues.jenkins-ci.org/browse/JENKINS-22734
> >> > > > <https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
> >> > > > specifically introduced in version 1.560.*
> >> > > > *It does not occur in previous versions and is fixed in versino 1.563.*
> >> > > >
> >> > > > So given that we currently use v1.455, I doubt that this JIRA is the one
> >> > > > creating the problem that we currently are experiencing.
> >> > > >
> >> > > > Chandrasekhar Gopal
> >> > > > Pivotal Hadoop -- Build, Release and Deployments
> >> > > > cgopal@gopivotal.com
> >> > > >
> >> > > >
> >> > > > On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >> > > >
> >> > > > > Julien,
> >> > > > >
> >> > > > > if you feel like helping to resolve this problem then I think I can give
> >> > > > > you
> >> > > > > the access to the system. Please reach out to me privately, so I can get
> >> > > > > your
> >> > > > > keys setup, etc.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Cos
> >> > > > >
> >> > > > > On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> >> > > > > > This looks to be this issue
> >> > > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> >> > > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
> >> > > > > access
> >> > > > > > to build slaves to actually get down and dirty with logs. Either way, our
> >> > > > > > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> >> > > > > > to talk to a few people here and see about updating our Jenkins setup and
> >> > > > > > any other work we need to do infra side to get this issue fixed. I'll
> >> > > > > > report back with anything I find so we can get a gameplan going.
> >> > > > > >
> >> > > > > > Julien
> >> > > > > >
> >> > > > > >
> >> > > > > > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks for checking it out, Mark. I've tried to untangle it last
> >> > > > > night, but
> >> > > > > > > without much success. The most weird part is that cleaning up build
> >> > > > > queue
> >> > > > > > > and
> >> > > > > > > restarting to the trunk-all job grinds to the same halt every time ;(
> >> > > > > > >
> >> > > > > > > Cos
> >> > > > > > >
> >> > > > > > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> >> > > > > > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
> >> > > > > don't
> >> > > > > > > > understand much about how the Jenkins artifact archiver works, I will
> >> > > > > > > poke
> >> > > > > > > > more).
> >> > > > > > > >
> >> > > > > > > > I don't have access to it, can someone with access to Jenkins master
> >> > > > > > > check
> >> > > > > > > > if life is good on it?
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
> >> > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
> >> > > > > disk
> >> > > > > > > > > space looks fine. I am trying to understand where the Jenkins
> >> > > > > archiver
> >> > > > > > > is
> >> > > > > > > > > archiving to, maybe that thing ran out of space...
> >> > > > > > > > > Mark
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
> >> > > > > julien.eid@cloudera.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> >> > > > > > > > >>
> >> > > > > > > > >> Fedora 18 box is blocking jobs from running for the past two days.
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > >
> >> > > > > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> >> > > > > > > > >> Stuck on archiving Sqoop artifacts.
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > >
> >> > > > >
> >
> >
Re:Re: Jenkins builds are frozen
Posted by pengwenwu <pe...@163.com>.
is possible network overloading cause failure as https://issues.jenkins-ci.org/browse/JENKINS-14332 ?
Regards,
Wenwu,Peng
At 2014-07-16 02:49:26, "Konstantin Boudnik" <co...@apache.org> wrote:
>Damn, centos6 slave still behaves - I think I will dive into this tomorrow to
>find out why the stuff like this
>
>http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Datafu/label=centos6/513/console
>
>is happening. I have already rebooted it - perhaps the timing was wrong and
>all these jobs failed because the slave was down? It is an unlikely
>explanation, but I can't come up with anything better atm ;(
>
>Cos
>
>On Mon, Jul 14, 2014 at 10:50PM, Konstantin Boudnik wrote:
>> Ok, Jenkins is at 1.572 now - let's see how the build proceeds from here.
>>
>> Cos
>>
>> On Mon, Jul 14, 2014 at 03:53PM, Konstantin Boudnik wrote:
>> > Damn, I can't: only my home computer's keys are added to the CI setup ;( Ok, I
>> > will do the upgrade in a couple of hours.
>> >
>> > On Mon, Jul 14, 2014 at 03:30PM, Konstantin Boudnik wrote:
>> > > Good one Chandrasekhar! Ok, Jenkins is completely stuck right now, so let me
>> > > start the upgrade right away and restart the server. We'll see how it goes
>> > > after.
>> > >
>> > > Cos
>> > >
>> > > On Mon, Jul 14, 2014 at 02:13PM, Chandrasekhar Gopal wrote:
>> > > > My 2 cents on Julien's comments about the OOM JIRAs. We had the same
>> > > > issue when we installed version 1.560.
>> > > >
>> > > > <> Copying artifacts from other build jobs would hang
>> > > > <> Slaves would disconnect at random.
>> > > >
>> > > > *However, as pointed out
>> > > > in https://issues.jenkins-ci.org/browse/JENKINS-22734
>> > > > <https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
>> > > > specifically introduced in version 1.560.*
>> > > > *It does not occur in previous versions and is fixed in versino 1.563.*
>> > > >
>> > > > So given that we currently use v1.455, I doubt that this JIRA is the one
>> > > > creating the problem that we currently are experiencing.
>> > > >
>> > > > Chandrasekhar Gopal
>> > > > Pivotal Hadoop -- Build, Release and Deployments
>> > > > cgopal@gopivotal.com
>> > > >
>> > > >
>> > > > On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> > > >
>> > > > > Julien,
>> > > > >
>> > > > > if you feel like helping to resolve this problem then I think I can give
>> > > > > you
>> > > > > the access to the system. Please reach out to me privately, so I can get
>> > > > > your
>> > > > > keys setup, etc.
>> > > > >
>> > > > > Thanks,
>> > > > > Cos
>> > > > >
>> > > > > On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
>> > > > > > This looks to be this issue
>> > > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
>> > > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
>> > > > > access
>> > > > > > to build slaves to actually get down and dirty with logs. Either way, our
>> > > > > > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
>> > > > > > to talk to a few people here and see about updating our Jenkins setup and
>> > > > > > any other work we need to do infra side to get this issue fixed. I'll
>> > > > > > report back with anything I find so we can get a gameplan going.
>> > > > > >
>> > > > > > Julien
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Thanks for checking it out, Mark. I've tried to untangle it last
>> > > > > night, but
>> > > > > > > without much success. The most weird part is that cleaning up build
>> > > > > queue
>> > > > > > > and
>> > > > > > > restarting to the trunk-all job grinds to the same halt every time ;(
>> > > > > > >
>> > > > > > > Cos
>> > > > > > >
>> > > > > > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
>> > > > > > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
>> > > > > don't
>> > > > > > > > understand much about how the Jenkins artifact archiver works, I will
>> > > > > > > poke
>> > > > > > > > more).
>> > > > > > > >
>> > > > > > > > I don't have access to it, can someone with access to Jenkins master
>> > > > > > > check
>> > > > > > > > if life is good on it?
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
>> > > > > disk
>> > > > > > > > > space looks fine. I am trying to understand where the Jenkins
>> > > > > archiver
>> > > > > > > is
>> > > > > > > > > archiving to, maybe that thing ran out of space...
>> > > > > > > > > Mark
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
>> > > > > julien.eid@cloudera.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
>> > > > > > > > >>
>> > > > > > > > >> Fedora 18 box is blocking jobs from running for the past two days.
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > >
>> > > > > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
>> > > > > > > > >> Stuck on archiving Sqoop artifacts.
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > >
>> > > > >
>
>
Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
Damn, centos6 slave still behaves - I think I will dive into this tomorrow to
find out why the stuff like this
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Datafu/label=centos6/513/console
is happening. I have already rebooted it - perhaps the timing was wrong and
all these jobs failed because the slave was down? It is an unlikely
explanation, but I can't come up with anything better atm ;(
Cos
On Mon, Jul 14, 2014 at 10:50PM, Konstantin Boudnik wrote:
> Ok, Jenkins is at 1.572 now - let's see how the build proceeds from here.
>
> Cos
>
> On Mon, Jul 14, 2014 at 03:53PM, Konstantin Boudnik wrote:
> > Damn, I can't: only my home computer's keys are added to the CI setup ;( Ok, I
> > will do the upgrade in a couple of hours.
> >
> > On Mon, Jul 14, 2014 at 03:30PM, Konstantin Boudnik wrote:
> > > Good one Chandrasekhar! Ok, Jenkins is completely stuck right now, so let me
> > > start the upgrade right away and restart the server. We'll see how it goes
> > > after.
> > >
> > > Cos
> > >
> > > On Mon, Jul 14, 2014 at 02:13PM, Chandrasekhar Gopal wrote:
> > > > My 2 cents on Julien's comments about the OOM JIRAs. We had the same
> > > > issue when we installed version 1.560.
> > > >
> > > > <> Copying artifacts from other build jobs would hang
> > > > <> Slaves would disconnect at random.
> > > >
> > > > *However, as pointed out
> > > > in https://issues.jenkins-ci.org/browse/JENKINS-22734
> > > > <https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
> > > > specifically introduced in version 1.560.*
> > > > *It does not occur in previous versions and is fixed in versino 1.563.*
> > > >
> > > > So given that we currently use v1.455, I doubt that this JIRA is the one
> > > > creating the problem that we currently are experiencing.
> > > >
> > > > Chandrasekhar Gopal
> > > > Pivotal Hadoop -- Build, Release and Deployments
> > > > cgopal@gopivotal.com
> > > >
> > > >
> > > > On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > > >
> > > > > Julien,
> > > > >
> > > > > if you feel like helping to resolve this problem then I think I can give
> > > > > you
> > > > > the access to the system. Please reach out to me privately, so I can get
> > > > > your
> > > > > keys setup, etc.
> > > > >
> > > > > Thanks,
> > > > > Cos
> > > > >
> > > > > On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> > > > > > This looks to be this issue
> > > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> > > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
> > > > > access
> > > > > > to build slaves to actually get down and dirty with logs. Either way, our
> > > > > > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> > > > > > to talk to a few people here and see about updating our Jenkins setup and
> > > > > > any other work we need to do infra side to get this issue fixed. I'll
> > > > > > report back with anything I find so we can get a gameplan going.
> > > > > >
> > > > > > Julien
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks for checking it out, Mark. I've tried to untangle it last
> > > > > night, but
> > > > > > > without much success. The most weird part is that cleaning up build
> > > > > queue
> > > > > > > and
> > > > > > > restarting to the trunk-all job grinds to the same halt every time ;(
> > > > > > >
> > > > > > > Cos
> > > > > > >
> > > > > > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > > > > > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
> > > > > don't
> > > > > > > > understand much about how the Jenkins artifact archiver works, I will
> > > > > > > poke
> > > > > > > > more).
> > > > > > > >
> > > > > > > > I don't have access to it, can someone with access to Jenkins master
> > > > > > > check
> > > > > > > > if life is good on it?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
> > > > > disk
> > > > > > > > > space looks fine. I am trying to understand where the Jenkins
> > > > > archiver
> > > > > > > is
> > > > > > > > > archiving to, maybe that thing ran out of space...
> > > > > > > > > Mark
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
> > > > > julien.eid@cloudera.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > > > > > > > >>
> > > > > > > > >> Fedora 18 box is blocking jobs from running for the past two days.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > >
> > > > > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > > > > > > > >> Stuck on archiving Sqoop artifacts.
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
Ok, Jenkins is at 1.572 now - let's see how the build proceeds from here.
Cos
On Mon, Jul 14, 2014 at 03:53PM, Konstantin Boudnik wrote:
> Damn, I can't: only my home computer's keys are added to the CI setup ;( Ok, I
> will do the upgrade in a couple of hours.
>
> On Mon, Jul 14, 2014 at 03:30PM, Konstantin Boudnik wrote:
> > Good one Chandrasekhar! Ok, Jenkins is completely stuck right now, so let me
> > start the upgrade right away and restart the server. We'll see how it goes
> > after.
> >
> > Cos
> >
> > On Mon, Jul 14, 2014 at 02:13PM, Chandrasekhar Gopal wrote:
> > > My 2 cents on Julien's comments about the OOM JIRAs. We had the same
> > > issue when we installed version 1.560.
> > >
> > > <> Copying artifacts from other build jobs would hang
> > > <> Slaves would disconnect at random.
> > >
> > > *However, as pointed out
> > > in https://issues.jenkins-ci.org/browse/JENKINS-22734
> > > <https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
> > > specifically introduced in version 1.560.*
> > > *It does not occur in previous versions and is fixed in versino 1.563.*
> > >
> > > So given that we currently use v1.455, I doubt that this JIRA is the one
> > > creating the problem that we currently are experiencing.
> > >
> > > Chandrasekhar Gopal
> > > Pivotal Hadoop -- Build, Release and Deployments
> > > cgopal@gopivotal.com
> > >
> > >
> > > On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> > >
> > > > Julien,
> > > >
> > > > if you feel like helping to resolve this problem then I think I can give
> > > > you
> > > > the access to the system. Please reach out to me privately, so I can get
> > > > your
> > > > keys setup, etc.
> > > >
> > > > Thanks,
> > > > Cos
> > > >
> > > > On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> > > > > This looks to be this issue
> > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> > > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
> > > > access
> > > > > to build slaves to actually get down and dirty with logs. Either way, our
> > > > > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> > > > > to talk to a few people here and see about updating our Jenkins setup and
> > > > > any other work we need to do infra side to get this issue fixed. I'll
> > > > > report back with anything I find so we can get a gameplan going.
> > > > >
> > > > > Julien
> > > > >
> > > > >
> > > > > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Thanks for checking it out, Mark. I've tried to untangle it last
> > > > night, but
> > > > > > without much success. The most weird part is that cleaning up build
> > > > queue
> > > > > > and
> > > > > > restarting to the trunk-all job grinds to the same halt every time ;(
> > > > > >
> > > > > > Cos
> > > > > >
> > > > > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > > > > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
> > > > don't
> > > > > > > understand much about how the Jenkins artifact archiver works, I will
> > > > > > poke
> > > > > > > more).
> > > > > > >
> > > > > > > I don't have access to it, can someone with access to Jenkins master
> > > > > > check
> > > > > > > if life is good on it?
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
> > > > disk
> > > > > > > > space looks fine. I am trying to understand where the Jenkins
> > > > archiver
> > > > > > is
> > > > > > > > archiving to, maybe that thing ran out of space...
> > > > > > > > Mark
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
> > > > julien.eid@cloudera.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > > > > > > >>
> > > > > > > >> Fedora 18 box is blocking jobs from running for the past two days.
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > > > > > > >> Stuck on archiving Sqoop artifacts.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
Damn, I can't: only my home computer's keys are added to the CI setup ;( Ok, I
will do the upgrade in a couple of hours.
On Mon, Jul 14, 2014 at 03:30PM, Konstantin Boudnik wrote:
> Good one Chandrasekhar! Ok, Jenkins is completely stuck right now, so let me
> start the upgrade right away and restart the server. We'll see how it goes
> after.
>
> Cos
>
> On Mon, Jul 14, 2014 at 02:13PM, Chandrasekhar Gopal wrote:
> > My 2 cents on Julien's comments about the OOM JIRAs. We had the same
> > issue when we installed version 1.560.
> >
> > <> Copying artifacts from other build jobs would hang
> > <> Slaves would disconnect at random.
> >
> > *However, as pointed out
> > in https://issues.jenkins-ci.org/browse/JENKINS-22734
> > <https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
> > specifically introduced in version 1.560.*
> > *It does not occur in previous versions and is fixed in versino 1.563.*
> >
> > So given that we currently use v1.455, I doubt that this JIRA is the one
> > creating the problem that we currently are experiencing.
> >
> > Chandrasekhar Gopal
> > Pivotal Hadoop -- Build, Release and Deployments
> > cgopal@gopivotal.com
> >
> >
> > On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >
> > > Julien,
> > >
> > > if you feel like helping to resolve this problem then I think I can give
> > > you
> > > the access to the system. Please reach out to me privately, so I can get
> > > your
> > > keys setup, etc.
> > >
> > > Thanks,
> > > Cos
> > >
> > > On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> > > > This looks to be this issue
> > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> > > > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
> > > access
> > > > to build slaves to actually get down and dirty with logs. Either way, our
> > > > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> > > > to talk to a few people here and see about updating our Jenkins setup and
> > > > any other work we need to do infra side to get this issue fixed. I'll
> > > > report back with anything I find so we can get a gameplan going.
> > > >
> > > > Julien
> > > >
> > > >
> > > > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
> > > wrote:
> > > >
> > > > > Thanks for checking it out, Mark. I've tried to untangle it last
> > > night, but
> > > > > without much success. The most weird part is that cleaning up build
> > > queue
> > > > > and
> > > > > restarting to the trunk-all job grinds to the same halt every time ;(
> > > > >
> > > > > Cos
> > > > >
> > > > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > > > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
> > > don't
> > > > > > understand much about how the Jenkins artifact archiver works, I will
> > > > > poke
> > > > > > more).
> > > > > >
> > > > > > I don't have access to it, can someone with access to Jenkins master
> > > > > check
> > > > > > if life is good on it?
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
> > > disk
> > > > > > > space looks fine. I am trying to understand where the Jenkins
> > > archiver
> > > > > is
> > > > > > > archiving to, maybe that thing ran out of space...
> > > > > > > Mark
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
> > > julien.eid@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > > > > > >>
> > > > > > >> Fedora 18 box is blocking jobs from running for the past two days.
> > > > > > >>
> > > > > > >>
> > > > >
> > > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > > > > > >> Stuck on archiving Sqoop artifacts.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > >
> > >
Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
Good one Chandrasekhar! Ok, Jenkins is completely stuck right now, so let me
start the upgrade right away and restart the server. We'll see how it goes
after.
Cos
On Mon, Jul 14, 2014 at 02:13PM, Chandrasekhar Gopal wrote:
> My 2 cents on Julien's comments about the OOM JIRAs. We had the same
> issue when we installed version 1.560.
>
> <> Copying artifacts from other build jobs would hang
> <> Slaves would disconnect at random.
>
> *However, as pointed out
> in https://issues.jenkins-ci.org/browse/JENKINS-22734
> <https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
> specifically introduced in version 1.560.*
> *It does not occur in previous versions and is fixed in versino 1.563.*
>
> So given that we currently use v1.455, I doubt that this JIRA is the one
> creating the problem that we currently are experiencing.
>
> Chandrasekhar Gopal
> Pivotal Hadoop -- Build, Release and Deployments
> cgopal@gopivotal.com
>
>
> On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Julien,
> >
> > if you feel like helping to resolve this problem then I think I can give
> > you
> > the access to the system. Please reach out to me privately, so I can get
> > your
> > keys setup, etc.
> >
> > Thanks,
> > Cos
> >
> > On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> > > This looks to be this issue
> > > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> > > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
> > access
> > > to build slaves to actually get down and dirty with logs. Either way, our
> > > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> > > to talk to a few people here and see about updating our Jenkins setup and
> > > any other work we need to do infra side to get this issue fixed. I'll
> > > report back with anything I find so we can get a gameplan going.
> > >
> > > Julien
> > >
> > >
> > > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > >
> > > > Thanks for checking it out, Mark. I've tried to untangle it last
> > night, but
> > > > without much success. The most weird part is that cleaning up build
> > queue
> > > > and
> > > > restarting to the trunk-all job grinds to the same halt every time ;(
> > > >
> > > > Cos
> > > >
> > > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
> > don't
> > > > > understand much about how the Jenkins artifact archiver works, I will
> > > > poke
> > > > > more).
> > > > >
> > > > > I don't have access to it, can someone with access to Jenkins master
> > > > check
> > > > > if life is good on it?
> > > > >
> > > > >
> > > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
> > wrote:
> > > > >
> > > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
> > disk
> > > > > > space looks fine. I am trying to understand where the Jenkins
> > archiver
> > > > is
> > > > > > archiving to, maybe that thing ran out of space...
> > > > > > Mark
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
> > julien.eid@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > > > > >>
> > > > > >> Fedora 18 box is blocking jobs from running for the past two days.
> > > > > >>
> > > > > >>
> > > >
> > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > > > > >> Stuck on archiving Sqoop artifacts.
> > > > > >>
> > > > > >
> > > > > >
> > > >
> >
Re: Jenkins builds are frozen
Posted by Chandrasekhar Gopal <cg...@gopivotal.com>.
My 2 cents on Julien's comments about the OOM JIRAs. We had the same
issue when we installed version 1.560.
<> Copying artifacts from other build jobs would hang
<> Slaves would disconnect at random.
*However, as pointed out
in https://issues.jenkins-ci.org/browse/JENKINS-22734
<https://issues.jenkins-ci.org/browse/JENKINS-22734>, this problem was
specifically introduced in version 1.560.*
*It does not occur in previous versions and is fixed in versino 1.563.*
So given that we currently use v1.455, I doubt that this JIRA is the one
creating the problem that we currently are experiencing.
Chandrasekhar Gopal
Pivotal Hadoop -- Build, Release and Deployments
cgopal@gopivotal.com
On Mon, Jul 14, 2014 at 1:38 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Julien,
>
> if you feel like helping to resolve this problem then I think I can give
> you
> the access to the system. Please reach out to me privately, so I can get
> your
> keys setup, etc.
>
> Thanks,
> Cos
>
> On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> > This looks to be this issue
> > https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> > https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have
> access
> > to build slaves to actually get down and dirty with logs. Either way, our
> > Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> > to talk to a few people here and see about updating our Jenkins setup and
> > any other work we need to do infra side to get this issue fixed. I'll
> > report back with anything I find so we can get a gameplan going.
> >
> > Julien
> >
> >
> > On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > Thanks for checking it out, Mark. I've tried to untangle it last
> night, but
> > > without much success. The most weird part is that cleaning up build
> queue
> > > and
> > > restarting to the trunk-all job grinds to the same halt every time ;(
> > >
> > > Cos
> > >
> > > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry,
> don't
> > > > understand much about how the Jenkins artifact archiver works, I will
> > > poke
> > > > more).
> > > >
> > > > I don't have access to it, can someone with access to Jenkins master
> > > check
> > > > if life is good on it?
> > > >
> > > >
> > > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org>
> wrote:
> > > >
> > > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave,
> disk
> > > > > space looks fine. I am trying to understand where the Jenkins
> archiver
> > > is
> > > > > archiving to, maybe that thing ran out of space...
> > > > > Mark
> > > > >
> > > > >
> > > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <
> julien.eid@cloudera.com>
> > > > > wrote:
> > > > >
> > > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > > > >>
> > > > >> Fedora 18 box is blocking jobs from running for the past two days.
> > > > >>
> > > > >>
> > >
> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > > > >> Stuck on archiving Sqoop artifacts.
> > > > >>
> > > > >
> > > > >
> > >
>
Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
Julien,
if you feel like helping to resolve this problem then I think I can give you
the access to the system. Please reach out to me privately, so I can get your
keys setup, etc.
Thanks,
Cos
On Mon, Jul 14, 2014 at 12:04PM, Julien Eid wrote:
> This looks to be this issue
> https://issues.jenkins-ci.org/browse/JENKINS-22734 or
> https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have access
> to build slaves to actually get down and dirty with logs. Either way, our
> Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
> to talk to a few people here and see about updating our Jenkins setup and
> any other work we need to do infra side to get this issue fixed. I'll
> report back with anything I find so we can get a gameplan going.
>
> Julien
>
>
> On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > Thanks for checking it out, Mark. I've tried to untangle it last night, but
> > without much success. The most weird part is that cleaning up build queue
> > and
> > restarting to the trunk-all job grinds to the same halt every time ;(
> >
> > Cos
> >
> > On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > > Maybe, it's archiving the artifacts on the Jenkins master? (sorry, don't
> > > understand much about how the Jenkins artifact archiver works, I will
> > poke
> > > more).
> > >
> > > I don't have access to it, can someone with access to Jenkins master
> > check
> > > if life is good on it?
> > >
> > >
> > > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org> wrote:
> > >
> > > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave, disk
> > > > space looks fine. I am trying to understand where the Jenkins archiver
> > is
> > > > archiving to, maybe that thing ran out of space...
> > > > Mark
> > > >
> > > >
> > > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <ju...@cloudera.com>
> > > > wrote:
> > > >
> > > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > > >>
> > > >> Fedora 18 box is blocking jobs from running for the past two days.
> > > >>
> > > >>
> > http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > > >> Stuck on archiving Sqoop artifacts.
> > > >>
> > > >
> > > >
> >
Re: Jenkins builds are frozen
Posted by Julien Eid <ju...@cloudera.com>.
This looks to be this issue
https://issues.jenkins-ci.org/browse/JENKINS-22734 or
https://issues.jenkins-ci.org/browse/JENKINS-22734 but I don't have access
to build slaves to actually get down and dirty with logs. Either way, our
Jenkins setup is pretty old, latest is 1.572 and ours is 1.455. I'm going
to talk to a few people here and see about updating our Jenkins setup and
any other work we need to do infra side to get this issue fixed. I'll
report back with anything I find so we can get a gameplan going.
Julien
On Fri, Jul 11, 2014 at 6:08 PM, Konstantin Boudnik <co...@apache.org> wrote:
> Thanks for checking it out, Mark. I've tried to untangle it last night, but
> without much success. The most weird part is that cleaning up build queue
> and
> restarting to the trunk-all job grinds to the same halt every time ;(
>
> Cos
>
> On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> > Maybe, it's archiving the artifacts on the Jenkins master? (sorry, don't
> > understand much about how the Jenkins artifact archiver works, I will
> poke
> > more).
> >
> > I don't have access to it, can someone with access to Jenkins master
> check
> > if life is good on it?
> >
> >
> > On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org> wrote:
> >
> > > Yeah, I noticed that too. Looking into it, ssh'ing into the slave, disk
> > > space looks fine. I am trying to understand where the Jenkins archiver
> is
> > > archiving to, maybe that thing ran out of space...
> > > Mark
> > >
> > >
> > > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <ju...@cloudera.com>
> > > wrote:
> > >
> > >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> > >>
> > >> Fedora 18 box is blocking jobs from running for the past two days.
> > >>
> > >>
> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> > >> Stuck on archiving Sqoop artifacts.
> > >>
> > >
> > >
>
Re: Jenkins builds are frozen
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks for checking it out, Mark. I've tried to untangle it last night, but
without much success. The most weird part is that cleaning up build queue and
restarting to the trunk-all job grinds to the same halt every time ;(
Cos
On Fri, Jul 11, 2014 at 10:29AM, Mark Grover wrote:
> Maybe, it's archiving the artifacts on the Jenkins master? (sorry, don't
> understand much about how the Jenkins artifact archiver works, I will poke
> more).
>
> I don't have access to it, can someone with access to Jenkins master check
> if life is good on it?
>
>
> On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org> wrote:
>
> > Yeah, I noticed that too. Looking into it, ssh'ing into the slave, disk
> > space looks fine. I am trying to understand where the Jenkins archiver is
> > archiving to, maybe that thing ran out of space...
> > Mark
> >
> >
> > On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <ju...@cloudera.com>
> > wrote:
> >
> >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
> >>
> >> Fedora 18 box is blocking jobs from running for the past two days.
> >>
> >> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> >> Stuck on archiving Sqoop artifacts.
> >>
> >
> >
Re: Jenkins builds are frozen
Posted by Mark Grover <ma...@apache.org>.
Maybe, it's archiving the artifacts on the Jenkins master? (sorry, don't
understand much about how the Jenkins artifact archiver works, I will poke
more).
I don't have access to it, can someone with access to Jenkins master check
if life is good on it?
On Fri, Jul 11, 2014 at 10:25 AM, Mark Grover <ma...@apache.org> wrote:
> Yeah, I noticed that too. Looking into it, ssh'ing into the slave, disk
> space looks fine. I am trying to understand where the Jenkins archiver is
> archiving to, maybe that thing ran out of space...
> Mark
>
>
> On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <ju...@cloudera.com>
> wrote:
>
>> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
>>
>> Fedora 18 box is blocking jobs from running for the past two days.
>>
>> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
>> Stuck on archiving Sqoop artifacts.
>>
>
>
Re: Jenkins builds are frozen
Posted by Mark Grover <ma...@apache.org>.
Yeah, I noticed that too. Looking into it, ssh'ing into the slave, disk
space looks fine. I am trying to understand where the Jenkins archiver is
archiving to, maybe that thing ran out of space...
Mark
On Fri, Jul 11, 2014 at 10:24 AM, Julien Eid <ju...@cloudera.com>
wrote:
> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Hadoop/
>
> Fedora 18 box is blocking jobs from running for the past two days.
>
> http://bigtop01.cloudera.org:8080/job/Bigtop-trunk-Sqoop/label=fedora18/624/console
> Stuck on archiving Sqoop artifacts.
>