You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Gavin McDonald <ga...@16degrees.com.au> on 2018/07/23 04:22:45 UTC
HBase nightly job failing forever
https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
can someone take a look into this, the job isnt much good if it is failing
all the time and even worse if it is being ignored.
Otherwise I'll disable the job in a dew days to release these wasted
resources
to builds that matter.
--
---
Gav...
Re: HBase nightly job failing forever
Posted by Sean Busbey <bu...@cloudera.com.INVALID>.
-builds@
this is perhaps good incentive for us to move forward on
branch-specific flaky tests so that we can move these out of the
nightly build automagically. (which is getting tracked in HBASE-20387)
Maybe in a week or three I'd have time to do it myself (and I gotta
get HBASE-20834 done first). Can help with pointers if someone else
has time earlier.
On Tue, Jul 24, 2018 at 10:58 AM, Josh Elser <el...@apache.org> wrote:
> Yep, sadly this is a very long tent-pole for us. There are many involved who
> have invested countless hours in making this better.
>
> Specific to that job you linked earlier, 3 test failures out of our total
> 4958 tests (0.06% failure rate) is all but "green" in my mind. I would ask
> that you keep that in mind, too.
>
> To that extent, others have also built another job specifically to find
> tests which are failing intermittently:
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
> I mention this as evidence to prove to you that this is not a baseless
> request from the HBase PMC ;)
>
>
> On 7/24/18 3:14 AM, Gav wrote:
>>
>> Ok, good enough, will wait, please also note 'master' branch and a few
>> others have been failing for over a month also.
>>
>> I will check in again next month to see how things are progressing
>>
>> Thanks
>>
>> Gav...
>>
>>
>> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>
>>> Hi Gav,
>>>
>>> Looking at the most recent results, I see that the job failed because of
>>> two unit test failures. These are something that will be looked at prior
>>> to the next 1.4.x release which is about to get off the ground.
>>>
>>> I'd kindly request that you not disable the job. Thanks for trying to
>>> find extra resources on these nodes.
>>>
>>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>>>
>>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>>>
>>>> can someone take a look into this, the job isnt much good if it is
>>>
>>> failing
>>>>
>>>> all the time and even worse if it is being ignored.
>>>>
>>>> Otherwise I'll disable the job in a dew days to release these wasted
>>>> resources
>>>> to builds that matter.
>>>>
>>>>
>>>
>>
>>
>
--
busbey
Re: HBase nightly job failing forever
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jul 25, 2018, at 10:48 AM, Chris Lambertus <cm...@apache.org> wrote:
>
> On-demand resources are certainly being considered (and we had these in the past,) but I will point out that ephemeral (“on-demand”) cloud builds are in direct opposition to some of the points brought up by Allen in the other jenkins storage thread, in that they tend to rely on persistent object storage in their workspaces to improve the efficiency of their builds. Perhaps this would be less of an issue with an on-demand instance which would theoretically have no resource contention?
Likely.
A lot of work went into greatly reducing the amount of time Hadoop spent in the build queue and running on the nodes. It was “the big one” but I feel like that’s not so true or at least harder to prove anymore. I estimate we shaved days off of the queue from 5 years ago. Part of that was keeping caches, since the ‘Hadoop’ queue nodes were large. But I feel like significantly more work went into “reducing the stupidity” out of the CI jobs though.
Two examples:
* For source changes, only building and unit testing the relevant parts of a patch. e.g., a patch that changes code in module A should only see module A’s unit tests run. Let the nightlies sort out any inter-module brokenness post-commit.
* if a patch is for documentation, only run mvn site. If a patch is for shell code, only run shellcheck and relevant unit tests. Running the java unit tests is pointless.
Building everything every time is a waste of time for modularized source trees.
Combined with the walls put up around the docker containers (e.g., limiting how many processes can be launched at one time, memory limits, etc), I personally felt much better that, other than disk space, the Hadoop jobs were being exemplary citizens vs. pre-Yetus.
Re: HBase nightly job failing forever
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jul 25, 2018, at 10:48 AM, Chris Lambertus <cm...@apache.org> wrote:
>
> On-demand resources are certainly being considered (and we had these in the past,) but I will point out that ephemeral (“on-demand”) cloud builds are in direct opposition to some of the points brought up by Allen in the other jenkins storage thread, in that they tend to rely on persistent object storage in their workspaces to improve the efficiency of their builds. Perhaps this would be less of an issue with an on-demand instance which would theoretically have no resource contention?
Likely.
A lot of work went into greatly reducing the amount of time Hadoop spent in the build queue and running on the nodes. It was “the big one” but I feel like that’s not so true or at least harder to prove anymore. I estimate we shaved days off of the queue from 5 years ago. Part of that was keeping caches, since the ‘Hadoop’ queue nodes were large. But I feel like significantly more work went into “reducing the stupidity” out of the CI jobs though.
Two examples:
* For source changes, only building and unit testing the relevant parts of a patch. e.g., a patch that changes code in module A should only see module A’s unit tests run. Let the nightlies sort out any inter-module brokenness post-commit.
* if a patch is for documentation, only run mvn site. If a patch is for shell code, only run shellcheck and relevant unit tests. Running the java unit tests is pointless.
Building everything every time is a waste of time for modularized source trees.
Combined with the walls put up around the docker containers (e.g., limiting how many processes can be launched at one time, memory limits, etc), I personally felt much better that, other than disk space, the Hadoop jobs were being exemplary citizens vs. pre-Yetus.
Re: HBase nightly job failing forever
Posted by Chris Lambertus <cm...@apache.org>.
> On Jul 25, 2018, at 10:34 AM, Andrew Purtell <ap...@apache.org> wrote:
>
> public clouds instead. I'm not sure if the ASF is set up to manage on
> demand billing for test resources but this could be advantageous. It would
> track actual usage not fixed costs. To avoid budget overrun there would be
> caps and limits. Eventually demand would hit this new ceiling but the
On-demand resources are certainly being considered (and we had these in the past,) but I will point out that ephemeral (“on-demand”) cloud builds are in direct opposition to some of the points brought up by Allen in the other jenkins storage thread, in that they tend to rely on persistent object storage in their workspaces to improve the efficiency of their builds. Perhaps this would be less of an issue with an on-demand instance which would theoretically have no resource contention?
-Chris
ASF Infra
> --
> Best regards,
> Andrew
Re: HBase nightly job failing forever
Posted by Chris Lambertus <cm...@apache.org>.
> On Jul 25, 2018, at 10:34 AM, Andrew Purtell <ap...@apache.org> wrote:
>
> public clouds instead. I'm not sure if the ASF is set up to manage on
> demand billing for test resources but this could be advantageous. It would
> track actual usage not fixed costs. To avoid budget overrun there would be
> caps and limits. Eventually demand would hit this new ceiling but the
On-demand resources are certainly being considered (and we had these in the past,) but I will point out that ephemeral (“on-demand”) cloud builds are in direct opposition to some of the points brought up by Allen in the other jenkins storage thread, in that they tend to rely on persistent object storage in their workspaces to improve the efficiency of their builds. Perhaps this would be less of an issue with an on-demand instance which would theoretically have no resource contention?
-Chris
ASF Infra
> --
> Best regards,
> Andrew
Re: HBase nightly job failing forever
Posted by Andrew Purtell <ap...@apache.org>.
Thanks Joan and Bertrand.
> The number of failed builds in our stream that are directly related to
this "tragedy of the commons" far exceeds the number of successful builds
at this point, and unfortunately Travis CI is having parallel capacity
issues that prevent us from moving to them wholesale as well.
This has been my experience. So at one point years ago I moved my work off
the shared pool at the ASF as an individual contributor and have been
funding the testing I personally do up on EC2 out of pocket. This isn't a
general solution for our project, though, as it depends on my time and
ability to contribute, and focuses only on what I'm doing at the moment
maybe not what the project would like to see happen most.
I will look into targeted donation at my employer but am not optimistic.
It might be better to look at decommissioning some if not most of the
overutilized fixed test resources and use on demand executors launched on
public clouds instead. I'm not sure if the ASF is set up to manage on
demand billing for test resources but this could be advantageous. It would
track actual usage not fixed costs. To avoid budget overrun there would be
caps and limits. Eventually demand would hit this new ceiling but the
impact would be longer queue waiting times not job failures due to
environmental stress, so that would be an improvement. Each job would run
in its own virtual server or container so would be free of many of the
environmental issues we see now. Or to get the same improvement on the
resources we have now limit executor parallelism. Better to have a job wait
in queue than to run and fail anyway because the host environment is under
stress. For what it's worth.
On Wed, Jul 25, 2018 at 10:20 AM Joan Touzet <jo...@lrtw.org> wrote:
> I'll speak to CouchDB - the donation is directly in the form of a Jenkins
> build agent with our tag, no money is changed hands. The donator received
> a letter from fundraising@a.o allowing for tax deduction on the equivalent
> amount that the ASF leasing the machine would have cost for a year's
> donation. We have 24x7 support on the node from the provider, who performs
> all sysadmin (rather than burdening Infra with having to run puppet on our
> build machine). This was arranged so we could have a FreeBSD node in the
> build array.
>
> We have another donator in the wings who will be adding a build node for
> us; at that point, we expect to move all of our builds to our own Jenkins
> build agents and won't be in the common pool any longer. The number of
> failed builds in our stream that are directly related to this "tragedy of
> the commons" far exceeds the number of successful builds at this point,
> and unfortunately Travis CI is having parallel capacity issues that prevent
> us from moving to them wholesale as well.
>
> -Joan
>
> ----- Original Message -----
> From: "Andrew Purtell" <ap...@apache.org>
> To: ipv6guru@gmail.com
> Cc: "Andrew Purtell" <ap...@apache.org>, "dev" <de...@hbase.apache.org>,
> builds@apache.org
> Sent: Wednesday, July 25, 2018 12:22:08 PM
> Subject: Re: HBase nightly job failing forever
>
> How does a targeted hardware donation work? I was under the impression that
> targeted donations are not accepted by the ASF. Maybe it is different in
> infrastructure, but this is the first time I've heard of it. Who does the
> donation on those projects? DataStax for Cassandra? Who for CouchDB? Google
> for Beam? By what process are the donations made and how are they audited
> to confirm the donation is spent on the desired resources? Can we get a
> contact for one of them for testimonial regarding this process? Is this
> process documented?
>
>
>
>
> On Tue, Jul 24, 2018 at 4:27 PM Gav <ip...@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >> Thanks for this note.
> >>
> >> I'm release managing the 1.4 release. I have been running the unit test
> >> suite on reasonably endowed EC2 instances and there are no observed
> always
> >> failing tests. A few can be flaky. In comparison the Apache test
> resources
> >> have been heavily resource constrained for years and frequently suffer
> from
> >> environmental effects like botched settings, disk space issues, and
> >> contention with other test executors.
> >>
> >
> > Our Jenkins nodes are configured via puppet these days and are pretty
> > stable, to which settings do you know of that might (still) be botched?
> > Yes, resources are shared and on occasion run to capacity. This is one
> > reason for my initial mail - these HBase builds are consuming 10 or more
> > executors
> > -at the same time- and are starving executors for other builds. The fact
> > these tests have been failing for well over a month and that you mention
> > below will be
> > ignoring them does not make for good cross ASF community spirit, we are
> > all in this together and every little bit helps. This is not a target at
> > one project, others
> > will be getting a similar note and I hope we can come to a resolution
> > suitable for all.
> > Disk space issues , yes, not on most of the Hadoop and related projects
> > nodes - H0-H12 do not have disk space issues. As a Hadoop related project
> > HBase should really be concentrating its builds there.
> >
> >
> >> I think a 1.4 release will happen regardless of the job test results on
> >> Apache infrastructure. I tend to ignore them as noisy and low signal.
> >> Others in the HBase community don't necessarily feel the same, so please
> >> don't take my viewpoint as particularly representative. We could try
> Alan's
> >> suggestion first, before ignoring them outright.
> >>
> >
> > No problem
> >
> >
> >> Has anyone given thought toward expanding the pool of test build
> >> resources? Or roping in cloud instances on demand? Jenkins has support
> for
> >> that.
> >>
> >
> > We have currently 19 Hadoop specific nodes available H0-H19 and another
> 28
> > or so general use 'ubuntu' nodes for all to use. In addition we have
> > projects
> > that have targetted donated resources and the likes of Cassandra, CouchDB
> > and Beam all have multiple nodes on which they have priority. I'll throw
> an
> > idea
> > out there than perhaps HBase could do something similar to increase our
> > node pool and at the same time have priority on a few nodes f their own
> via
> > a targeted
> > hardware donation.
> > Cloud on demand has been tried a year or two ago, we will revisit this
> > also soon.
> >
> > Summary then, we currently have over 80 nodes connected to our Jenkins
> > master - what figure did you have in mind when you say 'expanding the
> pool
> > of test build resources' ?
> >
> > Thanks
> >
> > Gav...
> >
> >
> >>
> >> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
> >> <aw...@effectivemachines.com.invalid> wrote:
> >>
> >>> I suspect the bigger issue is that the hbase tests are running
> >>> on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces,
> the
> >>> hbase tests are eating a significant majority of it and likely could be
> >>> dying randomly due to space issues. [All the hbase workspace
> directories +
> >>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
> >>> Significantly more than the 50 or so other jobs that run on those
> >>> machines.]
> >>>
> >>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the
> big
> >>> jobs to consume….
> >>>
> >>>
> >>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
> >>> >
> >>> > Yep, sadly this is a very long tent-pole for us. There are many
> >>> involved who have invested countless hours in making this better.
> >>> >
> >>> > Specific to that job you linked earlier, 3 test failures out of our
> >>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
> >>> would ask that you keep that in mind, too.
> >>> >
> >>> > To that extent, others have also built another job specifically to
> >>> find tests which are failing intermittently:
> >>>
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html
> .
> >>> I mention this as evidence to prove to you that this is not a baseless
> >>> request from the HBase PMC ;)
> >>> >
> >>> > On 7/24/18 3:14 AM, Gav wrote:
> >>> >> Ok, good enough, will wait, please also note 'master' branch and a
> few
> >>> >> others have been failing for over a month also.
> >>> >> I will check in again next month to see how things are progressing
> >>> >> Thanks
> >>> >> Gav...
> >>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org>
> wrote:
> >>> >>> Hi Gav,
> >>> >>>
> >>> >>> Looking at the most recent results, I see that the job failed
> >>> because of
> >>> >>> two unit test failures. These are something that will be looked at
> >>> prior
> >>> >>> to the next 1.4.x release which is about to get off the ground.
> >>> >>>
> >>> >>> I'd kindly request that you not disable the job. Thanks for trying
> to
> >>> >>> find extra resources on these nodes.
> >>> >>>
> >>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
> >>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
> >>> >>>>
> >>> >>>> can someone take a look into this, the job isnt much good if it is
> >>> >>> failing
> >>> >>>> all the time and even worse if it is being ignored.
> >>> >>>>
> >>> >>>> Otherwise I'll disable the job in a dew days to release these
> wasted
> >>> >>>> resources
> >>> >>>> to builds that matter.
> >>> >>>>
> >>> >>>>
> >>> >>>
> >>>
> >>>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Words like orphans lost among the crosstalk, meaning torn from truth's
> >> decrepit hands
> >> - A23, Crosstalk
> >>
> >
> >
> > --
> > Gav...
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Andrew Purtell <ap...@apache.org>.
Thanks Joan and Bertrand.
> The number of failed builds in our stream that are directly related to
this "tragedy of the commons" far exceeds the number of successful builds
at this point, and unfortunately Travis CI is having parallel capacity
issues that prevent us from moving to them wholesale as well.
This has been my experience. So at one point years ago I moved my work off
the shared pool at the ASF as an individual contributor and have been
funding the testing I personally do up on EC2 out of pocket. This isn't a
general solution for our project, though, as it depends on my time and
ability to contribute, and focuses only on what I'm doing at the moment
maybe not what the project would like to see happen most.
I will look into targeted donation at my employer but am not optimistic.
It might be better to look at decommissioning some if not most of the
overutilized fixed test resources and use on demand executors launched on
public clouds instead. I'm not sure if the ASF is set up to manage on
demand billing for test resources but this could be advantageous. It would
track actual usage not fixed costs. To avoid budget overrun there would be
caps and limits. Eventually demand would hit this new ceiling but the
impact would be longer queue waiting times not job failures due to
environmental stress, so that would be an improvement. Each job would run
in its own virtual server or container so would be free of many of the
environmental issues we see now. Or to get the same improvement on the
resources we have now limit executor parallelism. Better to have a job wait
in queue than to run and fail anyway because the host environment is under
stress. For what it's worth.
On Wed, Jul 25, 2018 at 10:20 AM Joan Touzet <jo...@lrtw.org> wrote:
> I'll speak to CouchDB - the donation is directly in the form of a Jenkins
> build agent with our tag, no money is changed hands. The donator received
> a letter from fundraising@a.o allowing for tax deduction on the equivalent
> amount that the ASF leasing the machine would have cost for a year's
> donation. We have 24x7 support on the node from the provider, who performs
> all sysadmin (rather than burdening Infra with having to run puppet on our
> build machine). This was arranged so we could have a FreeBSD node in the
> build array.
>
> We have another donator in the wings who will be adding a build node for
> us; at that point, we expect to move all of our builds to our own Jenkins
> build agents and won't be in the common pool any longer. The number of
> failed builds in our stream that are directly related to this "tragedy of
> the commons" far exceeds the number of successful builds at this point,
> and unfortunately Travis CI is having parallel capacity issues that prevent
> us from moving to them wholesale as well.
>
> -Joan
>
> ----- Original Message -----
> From: "Andrew Purtell" <ap...@apache.org>
> To: ipv6guru@gmail.com
> Cc: "Andrew Purtell" <ap...@apache.org>, "dev" <de...@hbase.apache.org>,
> builds@apache.org
> Sent: Wednesday, July 25, 2018 12:22:08 PM
> Subject: Re: HBase nightly job failing forever
>
> How does a targeted hardware donation work? I was under the impression that
> targeted donations are not accepted by the ASF. Maybe it is different in
> infrastructure, but this is the first time I've heard of it. Who does the
> donation on those projects? DataStax for Cassandra? Who for CouchDB? Google
> for Beam? By what process are the donations made and how are they audited
> to confirm the donation is spent on the desired resources? Can we get a
> contact for one of them for testimonial regarding this process? Is this
> process documented?
>
>
>
>
> On Tue, Jul 24, 2018 at 4:27 PM Gav <ip...@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >> Thanks for this note.
> >>
> >> I'm release managing the 1.4 release. I have been running the unit test
> >> suite on reasonably endowed EC2 instances and there are no observed
> always
> >> failing tests. A few can be flaky. In comparison the Apache test
> resources
> >> have been heavily resource constrained for years and frequently suffer
> from
> >> environmental effects like botched settings, disk space issues, and
> >> contention with other test executors.
> >>
> >
> > Our Jenkins nodes are configured via puppet these days and are pretty
> > stable, to which settings do you know of that might (still) be botched?
> > Yes, resources are shared and on occasion run to capacity. This is one
> > reason for my initial mail - these HBase builds are consuming 10 or more
> > executors
> > -at the same time- and are starving executors for other builds. The fact
> > these tests have been failing for well over a month and that you mention
> > below will be
> > ignoring them does not make for good cross ASF community spirit, we are
> > all in this together and every little bit helps. This is not a target at
> > one project, others
> > will be getting a similar note and I hope we can come to a resolution
> > suitable for all.
> > Disk space issues , yes, not on most of the Hadoop and related projects
> > nodes - H0-H12 do not have disk space issues. As a Hadoop related project
> > HBase should really be concentrating its builds there.
> >
> >
> >> I think a 1.4 release will happen regardless of the job test results on
> >> Apache infrastructure. I tend to ignore them as noisy and low signal.
> >> Others in the HBase community don't necessarily feel the same, so please
> >> don't take my viewpoint as particularly representative. We could try
> Alan's
> >> suggestion first, before ignoring them outright.
> >>
> >
> > No problem
> >
> >
> >> Has anyone given thought toward expanding the pool of test build
> >> resources? Or roping in cloud instances on demand? Jenkins has support
> for
> >> that.
> >>
> >
> > We have currently 19 Hadoop specific nodes available H0-H19 and another
> 28
> > or so general use 'ubuntu' nodes for all to use. In addition we have
> > projects
> > that have targetted donated resources and the likes of Cassandra, CouchDB
> > and Beam all have multiple nodes on which they have priority. I'll throw
> an
> > idea
> > out there than perhaps HBase could do something similar to increase our
> > node pool and at the same time have priority on a few nodes f their own
> via
> > a targeted
> > hardware donation.
> > Cloud on demand has been tried a year or two ago, we will revisit this
> > also soon.
> >
> > Summary then, we currently have over 80 nodes connected to our Jenkins
> > master - what figure did you have in mind when you say 'expanding the
> pool
> > of test build resources' ?
> >
> > Thanks
> >
> > Gav...
> >
> >
> >>
> >> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
> >> <aw...@effectivemachines.com.invalid> wrote:
> >>
> >>> I suspect the bigger issue is that the hbase tests are running
> >>> on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces,
> the
> >>> hbase tests are eating a significant majority of it and likely could be
> >>> dying randomly due to space issues. [All the hbase workspace
> directories +
> >>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
> >>> Significantly more than the 50 or so other jobs that run on those
> >>> machines.]
> >>>
> >>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the
> big
> >>> jobs to consume….
> >>>
> >>>
> >>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
> >>> >
> >>> > Yep, sadly this is a very long tent-pole for us. There are many
> >>> involved who have invested countless hours in making this better.
> >>> >
> >>> > Specific to that job you linked earlier, 3 test failures out of our
> >>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
> >>> would ask that you keep that in mind, too.
> >>> >
> >>> > To that extent, others have also built another job specifically to
> >>> find tests which are failing intermittently:
> >>>
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html
> .
> >>> I mention this as evidence to prove to you that this is not a baseless
> >>> request from the HBase PMC ;)
> >>> >
> >>> > On 7/24/18 3:14 AM, Gav wrote:
> >>> >> Ok, good enough, will wait, please also note 'master' branch and a
> few
> >>> >> others have been failing for over a month also.
> >>> >> I will check in again next month to see how things are progressing
> >>> >> Thanks
> >>> >> Gav...
> >>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org>
> wrote:
> >>> >>> Hi Gav,
> >>> >>>
> >>> >>> Looking at the most recent results, I see that the job failed
> >>> because of
> >>> >>> two unit test failures. These are something that will be looked at
> >>> prior
> >>> >>> to the next 1.4.x release which is about to get off the ground.
> >>> >>>
> >>> >>> I'd kindly request that you not disable the job. Thanks for trying
> to
> >>> >>> find extra resources on these nodes.
> >>> >>>
> >>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
> >>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
> >>> >>>>
> >>> >>>> can someone take a look into this, the job isnt much good if it is
> >>> >>> failing
> >>> >>>> all the time and even worse if it is being ignored.
> >>> >>>>
> >>> >>>> Otherwise I'll disable the job in a dew days to release these
> wasted
> >>> >>>> resources
> >>> >>>> to builds that matter.
> >>> >>>>
> >>> >>>>
> >>> >>>
> >>>
> >>>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Words like orphans lost among the crosstalk, meaning torn from truth's
> >> decrepit hands
> >> - A23, Crosstalk
> >>
> >
> >
> > --
> > Gav...
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Joan Touzet <jo...@lrtw.org>.
I'll speak to CouchDB - the donation is directly in the form of a Jenkins
build agent with our tag, no money is changed hands. The donator received
a letter from fundraising@a.o allowing for tax deduction on the equivalent
amount that the ASF leasing the machine would have cost for a year's
donation. We have 24x7 support on the node from the provider, who performs
all sysadmin (rather than burdening Infra with having to run puppet on our
build machine). This was arranged so we could have a FreeBSD node in the
build array.
We have another donator in the wings who will be adding a build node for
us; at that point, we expect to move all of our builds to our own Jenkins
build agents and won't be in the common pool any longer. The number of
failed builds in our stream that are directly related to this "tragedy of
the commons" far exceeds the number of successful builds at this point,
and unfortunately Travis CI is having parallel capacity issues that prevent
us from moving to them wholesale as well.
-Joan
----- Original Message -----
From: "Andrew Purtell" <ap...@apache.org>
To: ipv6guru@gmail.com
Cc: "Andrew Purtell" <ap...@apache.org>, "dev" <de...@hbase.apache.org>, builds@apache.org
Sent: Wednesday, July 25, 2018 12:22:08 PM
Subject: Re: HBase nightly job failing forever
How does a targeted hardware donation work? I was under the impression that
targeted donations are not accepted by the ASF. Maybe it is different in
infrastructure, but this is the first time I've heard of it. Who does the
donation on those projects? DataStax for Cassandra? Who for CouchDB? Google
for Beam? By what process are the donations made and how are they audited
to confirm the donation is spent on the desired resources? Can we get a
contact for one of them for testimonial regarding this process? Is this
process documented?
On Tue, Jul 24, 2018 at 4:27 PM Gav <ip...@gmail.com> wrote:
> Hi Andrew,
>
> On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org>
> wrote:
>
>> Thanks for this note.
>>
>> I'm release managing the 1.4 release. I have been running the unit test
>> suite on reasonably endowed EC2 instances and there are no observed always
>> failing tests. A few can be flaky. In comparison the Apache test resources
>> have been heavily resource constrained for years and frequently suffer from
>> environmental effects like botched settings, disk space issues, and
>> contention with other test executors.
>>
>
> Our Jenkins nodes are configured via puppet these days and are pretty
> stable, to which settings do you know of that might (still) be botched?
> Yes, resources are shared and on occasion run to capacity. This is one
> reason for my initial mail - these HBase builds are consuming 10 or more
> executors
> -at the same time- and are starving executors for other builds. The fact
> these tests have been failing for well over a month and that you mention
> below will be
> ignoring them does not make for good cross ASF community spirit, we are
> all in this together and every little bit helps. This is not a target at
> one project, others
> will be getting a similar note and I hope we can come to a resolution
> suitable for all.
> Disk space issues , yes, not on most of the Hadoop and related projects
> nodes - H0-H12 do not have disk space issues. As a Hadoop related project
> HBase should really be concentrating its builds there.
>
>
>> I think a 1.4 release will happen regardless of the job test results on
>> Apache infrastructure. I tend to ignore them as noisy and low signal.
>> Others in the HBase community don't necessarily feel the same, so please
>> don't take my viewpoint as particularly representative. We could try Alan's
>> suggestion first, before ignoring them outright.
>>
>
> No problem
>
>
>> Has anyone given thought toward expanding the pool of test build
>> resources? Or roping in cloud instances on demand? Jenkins has support for
>> that.
>>
>
> We have currently 19 Hadoop specific nodes available H0-H19 and another 28
> or so general use 'ubuntu' nodes for all to use. In addition we have
> projects
> that have targetted donated resources and the likes of Cassandra, CouchDB
> and Beam all have multiple nodes on which they have priority. I'll throw an
> idea
> out there than perhaps HBase could do something similar to increase our
> node pool and at the same time have priority on a few nodes f their own via
> a targeted
> hardware donation.
> Cloud on demand has been tried a year or two ago, we will revisit this
> also soon.
>
> Summary then, we currently have over 80 nodes connected to our Jenkins
> master - what figure did you have in mind when you say 'expanding the pool
> of test build resources' ?
>
> Thanks
>
> Gav...
>
>
>>
>> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
>> <aw...@effectivemachines.com.invalid> wrote:
>>
>>> I suspect the bigger issue is that the hbase tests are running
>>> on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
>>> hbase tests are eating a significant majority of it and likely could be
>>> dying randomly due to space issues. [All the hbase workspace directories +
>>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
>>> Significantly more than the 50 or so other jobs that run on those
>>> machines.]
>>>
>>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
>>> jobs to consume….
>>>
>>>
>>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>>> >
>>> > Yep, sadly this is a very long tent-pole for us. There are many
>>> involved who have invested countless hours in making this better.
>>> >
>>> > Specific to that job you linked earlier, 3 test failures out of our
>>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
>>> would ask that you keep that in mind, too.
>>> >
>>> > To that extent, others have also built another job specifically to
>>> find tests which are failing intermittently:
>>> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
>>> I mention this as evidence to prove to you that this is not a baseless
>>> request from the HBase PMC ;)
>>> >
>>> > On 7/24/18 3:14 AM, Gav wrote:
>>> >> Ok, good enough, will wait, please also note 'master' branch and a few
>>> >> others have been failing for over a month also.
>>> >> I will check in again next month to see how things are progressing
>>> >> Thanks
>>> >> Gav...
>>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>> >>> Hi Gav,
>>> >>>
>>> >>> Looking at the most recent results, I see that the job failed
>>> because of
>>> >>> two unit test failures. These are something that will be looked at
>>> prior
>>> >>> to the next 1.4.x release which is about to get off the ground.
>>> >>>
>>> >>> I'd kindly request that you not disable the job. Thanks for trying to
>>> >>> find extra resources on these nodes.
>>> >>>
>>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>> >>>>
>>> >>>> can someone take a look into this, the job isnt much good if it is
>>> >>> failing
>>> >>>> all the time and even worse if it is being ignored.
>>> >>>>
>>> >>>> Otherwise I'll disable the job in a dew days to release these wasted
>>> >>>> resources
>>> >>>> to builds that matter.
>>> >>>>
>>> >>>>
>>> >>>
>>>
>>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
>
>
> --
> Gav...
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Wed, Jul 25, 2018 at 6:22 PM Andrew Purtell <ap...@apache.org> wrote:
> ...How does a targeted hardware donation work? I was under the impression that
> targeted donations are not accepted by the ASF....
This has changed, last year IIRC - there's a bit of information at
https://www.apache.org/foundation/contributing under "targeted sponsor
program".
I suppose fundraising@a.o is best for more specific questions.
Targeted sponsors are listed at http://www.apache.org/foundation/thanks.html
-Bertrand
Re: HBase nightly job failing forever
Posted by Joan Touzet <jo...@lrtw.org>.
I'll speak to CouchDB - the donation is directly in the form of a Jenkins
build agent with our tag, no money is changed hands. The donator received
a letter from fundraising@a.o allowing for tax deduction on the equivalent
amount that the ASF leasing the machine would have cost for a year's
donation. We have 24x7 support on the node from the provider, who performs
all sysadmin (rather than burdening Infra with having to run puppet on our
build machine). This was arranged so we could have a FreeBSD node in the
build array.
We have another donator in the wings who will be adding a build node for
us; at that point, we expect to move all of our builds to our own Jenkins
build agents and won't be in the common pool any longer. The number of
failed builds in our stream that are directly related to this "tragedy of
the commons" far exceeds the number of successful builds at this point,
and unfortunately Travis CI is having parallel capacity issues that prevent
us from moving to them wholesale as well.
-Joan
----- Original Message -----
From: "Andrew Purtell" <ap...@apache.org>
To: ipv6guru@gmail.com
Cc: "Andrew Purtell" <ap...@apache.org>, "dev" <de...@hbase.apache.org>, builds@apache.org
Sent: Wednesday, July 25, 2018 12:22:08 PM
Subject: Re: HBase nightly job failing forever
How does a targeted hardware donation work? I was under the impression that
targeted donations are not accepted by the ASF. Maybe it is different in
infrastructure, but this is the first time I've heard of it. Who does the
donation on those projects? DataStax for Cassandra? Who for CouchDB? Google
for Beam? By what process are the donations made and how are they audited
to confirm the donation is spent on the desired resources? Can we get a
contact for one of them for testimonial regarding this process? Is this
process documented?
On Tue, Jul 24, 2018 at 4:27 PM Gav <ip...@gmail.com> wrote:
> Hi Andrew,
>
> On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org>
> wrote:
>
>> Thanks for this note.
>>
>> I'm release managing the 1.4 release. I have been running the unit test
>> suite on reasonably endowed EC2 instances and there are no observed always
>> failing tests. A few can be flaky. In comparison the Apache test resources
>> have been heavily resource constrained for years and frequently suffer from
>> environmental effects like botched settings, disk space issues, and
>> contention with other test executors.
>>
>
> Our Jenkins nodes are configured via puppet these days and are pretty
> stable, to which settings do you know of that might (still) be botched?
> Yes, resources are shared and on occasion run to capacity. This is one
> reason for my initial mail - these HBase builds are consuming 10 or more
> executors
> -at the same time- and are starving executors for other builds. The fact
> these tests have been failing for well over a month and that you mention
> below will be
> ignoring them does not make for good cross ASF community spirit, we are
> all in this together and every little bit helps. This is not a target at
> one project, others
> will be getting a similar note and I hope we can come to a resolution
> suitable for all.
> Disk space issues , yes, not on most of the Hadoop and related projects
> nodes - H0-H12 do not have disk space issues. As a Hadoop related project
> HBase should really be concentrating its builds there.
>
>
>> I think a 1.4 release will happen regardless of the job test results on
>> Apache infrastructure. I tend to ignore them as noisy and low signal.
>> Others in the HBase community don't necessarily feel the same, so please
>> don't take my viewpoint as particularly representative. We could try Alan's
>> suggestion first, before ignoring them outright.
>>
>
> No problem
>
>
>> Has anyone given thought toward expanding the pool of test build
>> resources? Or roping in cloud instances on demand? Jenkins has support for
>> that.
>>
>
> We have currently 19 Hadoop specific nodes available H0-H19 and another 28
> or so general use 'ubuntu' nodes for all to use. In addition we have
> projects
> that have targetted donated resources and the likes of Cassandra, CouchDB
> and Beam all have multiple nodes on which they have priority. I'll throw an
> idea
> out there than perhaps HBase could do something similar to increase our
> node pool and at the same time have priority on a few nodes f their own via
> a targeted
> hardware donation.
> Cloud on demand has been tried a year or two ago, we will revisit this
> also soon.
>
> Summary then, we currently have over 80 nodes connected to our Jenkins
> master - what figure did you have in mind when you say 'expanding the pool
> of test build resources' ?
>
> Thanks
>
> Gav...
>
>
>>
>> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
>> <aw...@effectivemachines.com.invalid> wrote:
>>
>>> I suspect the bigger issue is that the hbase tests are running
>>> on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
>>> hbase tests are eating a significant majority of it and likely could be
>>> dying randomly due to space issues. [All the hbase workspace directories +
>>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
>>> Significantly more than the 50 or so other jobs that run on those
>>> machines.]
>>>
>>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
>>> jobs to consume….
>>>
>>>
>>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>>> >
>>> > Yep, sadly this is a very long tent-pole for us. There are many
>>> involved who have invested countless hours in making this better.
>>> >
>>> > Specific to that job you linked earlier, 3 test failures out of our
>>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
>>> would ask that you keep that in mind, too.
>>> >
>>> > To that extent, others have also built another job specifically to
>>> find tests which are failing intermittently:
>>> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
>>> I mention this as evidence to prove to you that this is not a baseless
>>> request from the HBase PMC ;)
>>> >
>>> > On 7/24/18 3:14 AM, Gav wrote:
>>> >> Ok, good enough, will wait, please also note 'master' branch and a few
>>> >> others have been failing for over a month also.
>>> >> I will check in again next month to see how things are progressing
>>> >> Thanks
>>> >> Gav...
>>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>> >>> Hi Gav,
>>> >>>
>>> >>> Looking at the most recent results, I see that the job failed
>>> because of
>>> >>> two unit test failures. These are something that will be looked at
>>> prior
>>> >>> to the next 1.4.x release which is about to get off the ground.
>>> >>>
>>> >>> I'd kindly request that you not disable the job. Thanks for trying to
>>> >>> find extra resources on these nodes.
>>> >>>
>>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>> >>>>
>>> >>>> can someone take a look into this, the job isnt much good if it is
>>> >>> failing
>>> >>>> all the time and even worse if it is being ignored.
>>> >>>>
>>> >>>> Otherwise I'll disable the job in a dew days to release these wasted
>>> >>>> resources
>>> >>>> to builds that matter.
>>> >>>>
>>> >>>>
>>> >>>
>>>
>>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
>
>
> --
> Gav...
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Wed, Jul 25, 2018 at 6:22 PM Andrew Purtell <ap...@apache.org> wrote:
> ...How does a targeted hardware donation work? I was under the impression that
> targeted donations are not accepted by the ASF....
This has changed, last year IIRC - there's a bit of information at
https://www.apache.org/foundation/contributing under "targeted sponsor
program".
I suppose fundraising@a.o is best for more specific questions.
Targeted sponsors are listed at http://www.apache.org/foundation/thanks.html
-Bertrand
Re: HBase nightly job failing forever
Posted by Andrew Purtell <ap...@apache.org>.
How does a targeted hardware donation work? I was under the impression that
targeted donations are not accepted by the ASF. Maybe it is different in
infrastructure, but this is the first time I've heard of it. Who does the
donation on those projects? DataStax for Cassandra? Who for CouchDB? Google
for Beam? By what process are the donations made and how are they audited
to confirm the donation is spent on the desired resources? Can we get a
contact for one of them for testimonial regarding this process? Is this
process documented?
On Tue, Jul 24, 2018 at 4:27 PM Gav <ip...@gmail.com> wrote:
> Hi Andrew,
>
> On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org>
> wrote:
>
>> Thanks for this note.
>>
>> I'm release managing the 1.4 release. I have been running the unit test
>> suite on reasonably endowed EC2 instances and there are no observed always
>> failing tests. A few can be flaky. In comparison the Apache test resources
>> have been heavily resource constrained for years and frequently suffer from
>> environmental effects like botched settings, disk space issues, and
>> contention with other test executors.
>>
>
> Our Jenkins nodes are configured via puppet these days and are pretty
> stable, to which settings do you know of that might (still) be botched?
> Yes, resources are shared and on occasion run to capacity. This is one
> reason for my initial mail - these HBase builds are consuming 10 or more
> executors
> -at the same time- and are starving executors for other builds. The fact
> these tests have been failing for well over a month and that you mention
> below will be
> ignoring them does not make for good cross ASF community spirit, we are
> all in this together and every little bit helps. This is not a target at
> one project, others
> will be getting a similar note and I hope we can come to a resolution
> suitable for all.
> Disk space issues , yes, not on most of the Hadoop and related projects
> nodes - H0-H12 do not have disk space issues. As a Hadoop related project
> HBase should really be concentrating its builds there.
>
>
>> I think a 1.4 release will happen regardless of the job test results on
>> Apache infrastructure. I tend to ignore them as noisy and low signal.
>> Others in the HBase community don't necessarily feel the same, so please
>> don't take my viewpoint as particularly representative. We could try Alan's
>> suggestion first, before ignoring them outright.
>>
>
> No problem
>
>
>> Has anyone given thought toward expanding the pool of test build
>> resources? Or roping in cloud instances on demand? Jenkins has support for
>> that.
>>
>
> We have currently 19 Hadoop specific nodes available H0-H19 and another 28
> or so general use 'ubuntu' nodes for all to use. In addition we have
> projects
> that have targetted donated resources and the likes of Cassandra, CouchDB
> and Beam all have multiple nodes on which they have priority. I'll throw an
> idea
> out there than perhaps HBase could do something similar to increase our
> node pool and at the same time have priority on a few nodes f their own via
> a targeted
> hardware donation.
> Cloud on demand has been tried a year or two ago, we will revisit this
> also soon.
>
> Summary then, we currently have over 80 nodes connected to our Jenkins
> master - what figure did you have in mind when you say 'expanding the pool
> of test build resources' ?
>
> Thanks
>
> Gav...
>
>
>>
>> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
>> <aw...@effectivemachines.com.invalid> wrote:
>>
>>> I suspect the bigger issue is that the hbase tests are running
>>> on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
>>> hbase tests are eating a significant majority of it and likely could be
>>> dying randomly due to space issues. [All the hbase workspace directories +
>>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
>>> Significantly more than the 50 or so other jobs that run on those
>>> machines.]
>>>
>>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
>>> jobs to consume….
>>>
>>>
>>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>>> >
>>> > Yep, sadly this is a very long tent-pole for us. There are many
>>> involved who have invested countless hours in making this better.
>>> >
>>> > Specific to that job you linked earlier, 3 test failures out of our
>>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
>>> would ask that you keep that in mind, too.
>>> >
>>> > To that extent, others have also built another job specifically to
>>> find tests which are failing intermittently:
>>> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
>>> I mention this as evidence to prove to you that this is not a baseless
>>> request from the HBase PMC ;)
>>> >
>>> > On 7/24/18 3:14 AM, Gav wrote:
>>> >> Ok, good enough, will wait, please also note 'master' branch and a few
>>> >> others have been failing for over a month also.
>>> >> I will check in again next month to see how things are progressing
>>> >> Thanks
>>> >> Gav...
>>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>> >>> Hi Gav,
>>> >>>
>>> >>> Looking at the most recent results, I see that the job failed
>>> because of
>>> >>> two unit test failures. These are something that will be looked at
>>> prior
>>> >>> to the next 1.4.x release which is about to get off the ground.
>>> >>>
>>> >>> I'd kindly request that you not disable the job. Thanks for trying to
>>> >>> find extra resources on these nodes.
>>> >>>
>>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>> >>>>
>>> >>>> can someone take a look into this, the job isnt much good if it is
>>> >>> failing
>>> >>>> all the time and even worse if it is being ignored.
>>> >>>>
>>> >>>> Otherwise I'll disable the job in a dew days to release these wasted
>>> >>>> resources
>>> >>>> to builds that matter.
>>> >>>>
>>> >>>>
>>> >>>
>>>
>>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
>
>
> --
> Gav...
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Jul 25, 2018 at 2:36 AM Robert Munteanu <ro...@apache.org> wrote:
> Hi,
>
> On Wed, 2018-07-25 at 09:27 +1000, Gav wrote:
> > Disk space issues , yes, not on most of the Hadoop and related
> > projects
> > nodes - H0-H12 do not have disk space issues. As a Hadoop related
> > project
> > HBase should really be concentrating its builds there.
>
> A suggestion from the sidelines. We could add a 'large-disk-space'
> label to the jobs that use a lot of disk space and then also attach it
> to the executors that offer a lot of disk space.
>
Sure ... though I think it is the small disk that is the odd-man-out. The
nodes usually have lots of space, but a some few don't. And we're also
provisioning with much larger disks nowadays.
Cheers,
-g
Re: HBase nightly job failing forever
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Jul 25, 2018 at 2:36 AM Robert Munteanu <ro...@apache.org> wrote:
> Hi,
>
> On Wed, 2018-07-25 at 09:27 +1000, Gav wrote:
> > Disk space issues , yes, not on most of the Hadoop and related
> > projects
> > nodes - H0-H12 do not have disk space issues. As a Hadoop related
> > project
> > HBase should really be concentrating its builds there.
>
> A suggestion from the sidelines. We could add a 'large-disk-space'
> label to the jobs that use a lot of disk space and then also attach it
> to the executors that offer a lot of disk space.
>
Sure ... though I think it is the small disk that is the odd-man-out. The
nodes usually have lots of space, but a some few don't. And we're also
provisioning with much larger disks nowadays.
Cheers,
-g
Re: HBase nightly job failing forever
Posted by Robert Munteanu <ro...@apache.org>.
Hi,
On Wed, 2018-07-25 at 09:27 +1000, Gav wrote:
> Disk space issues , yes, not on most of the Hadoop and related
> projects
> nodes - H0-H12 do not have disk space issues. As a Hadoop related
> project
> HBase should really be concentrating its builds there.
A suggestion from the sidelines. We could add a 'large-disk-space'
label to the jobs that use a lot of disk space and then also attach it
to the executors that offer a lot of disk space.
Robert
Re: HBase nightly job failing forever
Posted by Robert Munteanu <ro...@apache.org>.
Hi,
On Wed, 2018-07-25 at 09:27 +1000, Gav wrote:
> Disk space issues , yes, not on most of the Hadoop and related
> projects
> nodes - H0-H12 do not have disk space issues. As a Hadoop related
> project
> HBase should really be concentrating its builds there.
A suggestion from the sidelines. We could add a 'large-disk-space'
label to the jobs that use a lot of disk space and then also attach it
to the executors that offer a lot of disk space.
Robert
Re: HBase nightly job failing forever
Posted by Andrew Purtell <ap...@apache.org>.
How does a targeted hardware donation work? I was under the impression that
targeted donations are not accepted by the ASF. Maybe it is different in
infrastructure, but this is the first time I've heard of it. Who does the
donation on those projects? DataStax for Cassandra? Who for CouchDB? Google
for Beam? By what process are the donations made and how are they audited
to confirm the donation is spent on the desired resources? Can we get a
contact for one of them for testimonial regarding this process? Is this
process documented?
On Tue, Jul 24, 2018 at 4:27 PM Gav <ip...@gmail.com> wrote:
> Hi Andrew,
>
> On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org>
> wrote:
>
>> Thanks for this note.
>>
>> I'm release managing the 1.4 release. I have been running the unit test
>> suite on reasonably endowed EC2 instances and there are no observed always
>> failing tests. A few can be flaky. In comparison the Apache test resources
>> have been heavily resource constrained for years and frequently suffer from
>> environmental effects like botched settings, disk space issues, and
>> contention with other test executors.
>>
>
> Our Jenkins nodes are configured via puppet these days and are pretty
> stable, to which settings do you know of that might (still) be botched?
> Yes, resources are shared and on occasion run to capacity. This is one
> reason for my initial mail - these HBase builds are consuming 10 or more
> executors
> -at the same time- and are starving executors for other builds. The fact
> these tests have been failing for well over a month and that you mention
> below will be
> ignoring them does not make for good cross ASF community spirit, we are
> all in this together and every little bit helps. This is not a target at
> one project, others
> will be getting a similar note and I hope we can come to a resolution
> suitable for all.
> Disk space issues , yes, not on most of the Hadoop and related projects
> nodes - H0-H12 do not have disk space issues. As a Hadoop related project
> HBase should really be concentrating its builds there.
>
>
>> I think a 1.4 release will happen regardless of the job test results on
>> Apache infrastructure. I tend to ignore them as noisy and low signal.
>> Others in the HBase community don't necessarily feel the same, so please
>> don't take my viewpoint as particularly representative. We could try Alan's
>> suggestion first, before ignoring them outright.
>>
>
> No problem
>
>
>> Has anyone given thought toward expanding the pool of test build
>> resources? Or roping in cloud instances on demand? Jenkins has support for
>> that.
>>
>
> We have currently 19 Hadoop specific nodes available H0-H19 and another 28
> or so general use 'ubuntu' nodes for all to use. In addition we have
> projects
> that have targetted donated resources and the likes of Cassandra, CouchDB
> and Beam all have multiple nodes on which they have priority. I'll throw an
> idea
> out there than perhaps HBase could do something similar to increase our
> node pool and at the same time have priority on a few nodes f their own via
> a targeted
> hardware donation.
> Cloud on demand has been tried a year or two ago, we will revisit this
> also soon.
>
> Summary then, we currently have over 80 nodes connected to our Jenkins
> master - what figure did you have in mind when you say 'expanding the pool
> of test build resources' ?
>
> Thanks
>
> Gav...
>
>
>>
>> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
>> <aw...@effectivemachines.com.invalid> wrote:
>>
>>> I suspect the bigger issue is that the hbase tests are running
>>> on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
>>> hbase tests are eating a significant majority of it and likely could be
>>> dying randomly due to space issues. [All the hbase workspace directories +
>>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
>>> Significantly more than the 50 or so other jobs that run on those
>>> machines.]
>>>
>>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
>>> jobs to consume….
>>>
>>>
>>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>>> >
>>> > Yep, sadly this is a very long tent-pole for us. There are many
>>> involved who have invested countless hours in making this better.
>>> >
>>> > Specific to that job you linked earlier, 3 test failures out of our
>>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
>>> would ask that you keep that in mind, too.
>>> >
>>> > To that extent, others have also built another job specifically to
>>> find tests which are failing intermittently:
>>> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
>>> I mention this as evidence to prove to you that this is not a baseless
>>> request from the HBase PMC ;)
>>> >
>>> > On 7/24/18 3:14 AM, Gav wrote:
>>> >> Ok, good enough, will wait, please also note 'master' branch and a few
>>> >> others have been failing for over a month also.
>>> >> I will check in again next month to see how things are progressing
>>> >> Thanks
>>> >> Gav...
>>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>> >>> Hi Gav,
>>> >>>
>>> >>> Looking at the most recent results, I see that the job failed
>>> because of
>>> >>> two unit test failures. These are something that will be looked at
>>> prior
>>> >>> to the next 1.4.x release which is about to get off the ground.
>>> >>>
>>> >>> I'd kindly request that you not disable the job. Thanks for trying to
>>> >>> find extra resources on these nodes.
>>> >>>
>>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>> >>>>
>>> >>>> can someone take a look into this, the job isnt much good if it is
>>> >>> failing
>>> >>>> all the time and even worse if it is being ignored.
>>> >>>>
>>> >>>> Otherwise I'll disable the job in a dew days to release these wasted
>>> >>>> resources
>>> >>>> to builds that matter.
>>> >>>>
>>> >>>>
>>> >>>
>>>
>>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
>
>
> --
> Gav...
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Gav <ip...@gmail.com>.
Hi Andrew,
On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org> wrote:
> Thanks for this note.
>
> I'm release managing the 1.4 release. I have been running the unit test
> suite on reasonably endowed EC2 instances and there are no observed always
> failing tests. A few can be flaky. In comparison the Apache test resources
> have been heavily resource constrained for years and frequently suffer from
> environmental effects like botched settings, disk space issues, and
> contention with other test executors.
>
Our Jenkins nodes are configured via puppet these days and are pretty
stable, to which settings do you know of that might (still) be botched?
Yes, resources are shared and on occasion run to capacity. This is one
reason for my initial mail - these HBase builds are consuming 10 or more
executors
-at the same time- and are starving executors for other builds. The fact
these tests have been failing for well over a month and that you mention
below will be
ignoring them does not make for good cross ASF community spirit, we are all
in this together and every little bit helps. This is not a target at one
project, others
will be getting a similar note and I hope we can come to a resolution
suitable for all.
Disk space issues , yes, not on most of the Hadoop and related projects
nodes - H0-H12 do not have disk space issues. As a Hadoop related project
HBase should really be concentrating its builds there.
> I think a 1.4 release will happen regardless of the job test results on
> Apache infrastructure. I tend to ignore them as noisy and low signal.
> Others in the HBase community don't necessarily feel the same, so please
> don't take my viewpoint as particularly representative. We could try Alan's
> suggestion first, before ignoring them outright.
>
No problem
> Has anyone given thought toward expanding the pool of test build
> resources? Or roping in cloud instances on demand? Jenkins has support for
> that.
>
We have currently 19 Hadoop specific nodes available H0-H19 and another 28
or so general use 'ubuntu' nodes for all to use. In addition we have
projects
that have targetted donated resources and the likes of Cassandra, CouchDB
and Beam all have multiple nodes on which they have priority. I'll throw an
idea
out there than perhaps HBase could do something similar to increase our
node pool and at the same time have priority on a few nodes f their own via
a targeted
hardware donation.
Cloud on demand has been tried a year or two ago, we will revisit this also
soon.
Summary then, we currently have over 80 nodes connected to our Jenkins
master - what figure did you have in mind when you say 'expanding the pool
of test build resources' ?
Thanks
Gav...
>
> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
> <aw...@effectivemachines.com.invalid> wrote:
>
>> I suspect the bigger issue is that the hbase tests are running on
>> the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
>> hbase tests are eating a significant majority of it and likely could be
>> dying randomly due to space issues. [All the hbase workspace directories +
>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
>> Significantly more than the 50 or so other jobs that run on those
>> machines.]
>>
>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
>> jobs to consume….
>>
>>
>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>> >
>> > Yep, sadly this is a very long tent-pole for us. There are many
>> involved who have invested countless hours in making this better.
>> >
>> > Specific to that job you linked earlier, 3 test failures out of our
>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
>> would ask that you keep that in mind, too.
>> >
>> > To that extent, others have also built another job specifically to find
>> tests which are failing intermittently:
>> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
>> I mention this as evidence to prove to you that this is not a baseless
>> request from the HBase PMC ;)
>> >
>> > On 7/24/18 3:14 AM, Gav wrote:
>> >> Ok, good enough, will wait, please also note 'master' branch and a few
>> >> others have been failing for over a month also.
>> >> I will check in again next month to see how things are progressing
>> >> Thanks
>> >> Gav...
>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>> >>> Hi Gav,
>> >>>
>> >>> Looking at the most recent results, I see that the job failed because
>> of
>> >>> two unit test failures. These are something that will be looked at
>> prior
>> >>> to the next 1.4.x release which is about to get off the ground.
>> >>>
>> >>> I'd kindly request that you not disable the job. Thanks for trying to
>> >>> find extra resources on these nodes.
>> >>>
>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>> >>>>
>> >>>> can someone take a look into this, the job isnt much good if it is
>> >>> failing
>> >>>> all the time and even worse if it is being ignored.
>> >>>>
>> >>>> Otherwise I'll disable the job in a dew days to release these wasted
>> >>>> resources
>> >>>> to builds that matter.
>> >>>>
>> >>>>
>> >>>
>>
>>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
--
Gav...
Re: HBase nightly job failing forever
Posted by Gav <ip...@gmail.com>.
Hi Andrew,
On Wed, Jul 25, 2018 at 3:21 AM Andrew Purtell <ap...@apache.org> wrote:
> Thanks for this note.
>
> I'm release managing the 1.4 release. I have been running the unit test
> suite on reasonably endowed EC2 instances and there are no observed always
> failing tests. A few can be flaky. In comparison the Apache test resources
> have been heavily resource constrained for years and frequently suffer from
> environmental effects like botched settings, disk space issues, and
> contention with other test executors.
>
Our Jenkins nodes are configured via puppet these days and are pretty
stable, to which settings do you know of that might (still) be botched?
Yes, resources are shared and on occasion run to capacity. This is one
reason for my initial mail - these HBase builds are consuming 10 or more
executors
-at the same time- and are starving executors for other builds. The fact
these tests have been failing for well over a month and that you mention
below will be
ignoring them does not make for good cross ASF community spirit, we are all
in this together and every little bit helps. This is not a target at one
project, others
will be getting a similar note and I hope we can come to a resolution
suitable for all.
Disk space issues , yes, not on most of the Hadoop and related projects
nodes - H0-H12 do not have disk space issues. As a Hadoop related project
HBase should really be concentrating its builds there.
> I think a 1.4 release will happen regardless of the job test results on
> Apache infrastructure. I tend to ignore them as noisy and low signal.
> Others in the HBase community don't necessarily feel the same, so please
> don't take my viewpoint as particularly representative. We could try Alan's
> suggestion first, before ignoring them outright.
>
No problem
> Has anyone given thought toward expanding the pool of test build
> resources? Or roping in cloud instances on demand? Jenkins has support for
> that.
>
We have currently 19 Hadoop specific nodes available H0-H19 and another 28
or so general use 'ubuntu' nodes for all to use. In addition we have
projects
that have targetted donated resources and the likes of Cassandra, CouchDB
and Beam all have multiple nodes on which they have priority. I'll throw an
idea
out there than perhaps HBase could do something similar to increase our
node pool and at the same time have priority on a few nodes f their own via
a targeted
hardware donation.
Cloud on demand has been tried a year or two ago, we will revisit this also
soon.
Summary then, we currently have over 80 nodes connected to our Jenkins
master - what figure did you have in mind when you say 'expanding the pool
of test build resources' ?
Thanks
Gav...
>
> On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
> <aw...@effectivemachines.com.invalid> wrote:
>
>> I suspect the bigger issue is that the hbase tests are running on
>> the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
>> hbase tests are eating a significant majority of it and likely could be
>> dying randomly due to space issues. [All the hbase workspace directories +
>> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
>> Significantly more than the 50 or so other jobs that run on those
>> machines.]
>>
>> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
>> jobs to consume….
>>
>>
>> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>> >
>> > Yep, sadly this is a very long tent-pole for us. There are many
>> involved who have invested countless hours in making this better.
>> >
>> > Specific to that job you linked earlier, 3 test failures out of our
>> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
>> would ask that you keep that in mind, too.
>> >
>> > To that extent, others have also built another job specifically to find
>> tests which are failing intermittently:
>> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
>> I mention this as evidence to prove to you that this is not a baseless
>> request from the HBase PMC ;)
>> >
>> > On 7/24/18 3:14 AM, Gav wrote:
>> >> Ok, good enough, will wait, please also note 'master' branch and a few
>> >> others have been failing for over a month also.
>> >> I will check in again next month to see how things are progressing
>> >> Thanks
>> >> Gav...
>> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>> >>> Hi Gav,
>> >>>
>> >>> Looking at the most recent results, I see that the job failed because
>> of
>> >>> two unit test failures. These are something that will be looked at
>> prior
>> >>> to the next 1.4.x release which is about to get off the ground.
>> >>>
>> >>> I'd kindly request that you not disable the job. Thanks for trying to
>> >>> find extra resources on these nodes.
>> >>>
>> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>> >>>>
>> >>>> can someone take a look into this, the job isnt much good if it is
>> >>> failing
>> >>>> all the time and even worse if it is being ignored.
>> >>>>
>> >>>> Otherwise I'll disable the job in a dew days to release these wasted
>> >>>> resources
>> >>>> to builds that matter.
>> >>>>
>> >>>>
>> >>>
>>
>>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
--
Gav...
Re: HBase nightly job failing forever
Posted by Andrew Purtell <ap...@apache.org>.
Thanks for this note.
I'm release managing the 1.4 release. I have been running the unit test
suite on reasonably endowed EC2 instances and there are no observed always
failing tests. A few can be flaky. In comparison the Apache test resources
have been heavily resource constrained for years and frequently suffer from
environmental effects like botched settings, disk space issues, and
contention with other test executors. I think a 1.4 release will happen
regardless of the job test results on Apache infrastructure. I tend to
ignore them as noisy and low signal. Others in the HBase community don't
necessarily feel the same, so please don't take my viewpoint as
particularly representative. We could try Alan's suggestion first, before
ignoring them outright.
Has anyone given thought toward expanding the pool of test build resources?
Or roping in cloud instances on demand? Jenkins has support for that.
On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:
> I suspect the bigger issue is that the hbase tests are running on
> the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
> hbase tests are eating a significant majority of it and likely could be
> dying randomly due to space issues. [All the hbase workspace directories +
> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
> Significantly more than the 50 or so other jobs that run on those
> machines.]
>
> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
> jobs to consume….
>
>
> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
> >
> > Yep, sadly this is a very long tent-pole for us. There are many involved
> who have invested countless hours in making this better.
> >
> > Specific to that job you linked earlier, 3 test failures out of our
> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
> would ask that you keep that in mind, too.
> >
> > To that extent, others have also built another job specifically to find
> tests which are failing intermittently:
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
> I mention this as evidence to prove to you that this is not a baseless
> request from the HBase PMC ;)
> >
> > On 7/24/18 3:14 AM, Gav wrote:
> >> Ok, good enough, will wait, please also note 'master' branch and a few
> >> others have been failing for over a month also.
> >> I will check in again next month to see how things are progressing
> >> Thanks
> >> Gav...
> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
> >>> Hi Gav,
> >>>
> >>> Looking at the most recent results, I see that the job failed because
> of
> >>> two unit test failures. These are something that will be looked at
> prior
> >>> to the next 1.4.x release which is about to get off the ground.
> >>>
> >>> I'd kindly request that you not disable the job. Thanks for trying to
> >>> find extra resources on these nodes.
> >>>
> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
> >>>>
> >>>> can someone take a look into this, the job isnt much good if it is
> >>> failing
> >>>> all the time and even worse if it is being ignored.
> >>>>
> >>>> Otherwise I'll disable the job in a dew days to release these wasted
> >>>> resources
> >>>> to builds that matter.
> >>>>
> >>>>
> >>>
>
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Andrew Purtell <ap...@apache.org>.
Thanks for this note.
I'm release managing the 1.4 release. I have been running the unit test
suite on reasonably endowed EC2 instances and there are no observed always
failing tests. A few can be flaky. In comparison the Apache test resources
have been heavily resource constrained for years and frequently suffer from
environmental effects like botched settings, disk space issues, and
contention with other test executors. I think a 1.4 release will happen
regardless of the job test results on Apache infrastructure. I tend to
ignore them as noisy and low signal. Others in the HBase community don't
necessarily feel the same, so please don't take my viewpoint as
particularly representative. We could try Alan's suggestion first, before
ignoring them outright.
Has anyone given thought toward expanding the pool of test build resources?
Or roping in cloud instances on demand? Jenkins has support for that.
On Tue, Jul 24, 2018 at 9:16 AM Allen Wittenauer
<aw...@effectivemachines.com.invalid> wrote:
> I suspect the bigger issue is that the hbase tests are running on
> the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the
> hbase tests are eating a significant majority of it and likely could be
> dying randomly due to space issues. [All the hbase workspace directories +
> the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space.
> Significantly more than the 50 or so other jobs that run on those
> machines.]
>
> By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big
> jobs to consume….
>
>
> > On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
> >
> > Yep, sadly this is a very long tent-pole for us. There are many involved
> who have invested countless hours in making this better.
> >
> > Specific to that job you linked earlier, 3 test failures out of our
> total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
> would ask that you keep that in mind, too.
> >
> > To that extent, others have also built another job specifically to find
> tests which are failing intermittently:
> https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
> I mention this as evidence to prove to you that this is not a baseless
> request from the HBase PMC ;)
> >
> > On 7/24/18 3:14 AM, Gav wrote:
> >> Ok, good enough, will wait, please also note 'master' branch and a few
> >> others have been failing for over a month also.
> >> I will check in again next month to see how things are progressing
> >> Thanks
> >> Gav...
> >> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
> >>> Hi Gav,
> >>>
> >>> Looking at the most recent results, I see that the job failed because
> of
> >>> two unit test failures. These are something that will be looked at
> prior
> >>> to the next 1.4.x release which is about to get off the ground.
> >>>
> >>> I'd kindly request that you not disable the job. Thanks for trying to
> >>> find extra resources on these nodes.
> >>>
> >>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
> >>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
> >>>>
> >>>> can someone take a look into this, the job isnt much good if it is
> >>> failing
> >>>> all the time and even worse if it is being ignored.
> >>>>
> >>>> Otherwise I'll disable the job in a dew days to release these wasted
> >>>> resources
> >>>> to builds that matter.
> >>>>
> >>>>
> >>>
>
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: HBase nightly job failing forever
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
I suspect the bigger issue is that the hbase tests are running on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the hbase tests are eating a significant majority of it and likely could be dying randomly due to space issues. [All the hbase workspace directories + the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space. Significantly more than the 50 or so other jobs that run on those machines.]
By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big jobs to consume….
> On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>
> Yep, sadly this is a very long tent-pole for us. There are many involved who have invested countless hours in making this better.
>
> Specific to that job you linked earlier, 3 test failures out of our total 4958 tests (0.06% failure rate) is all but "green" in my mind. I would ask that you keep that in mind, too.
>
> To that extent, others have also built another job specifically to find tests which are failing intermittently: https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html. I mention this as evidence to prove to you that this is not a baseless request from the HBase PMC ;)
>
> On 7/24/18 3:14 AM, Gav wrote:
>> Ok, good enough, will wait, please also note 'master' branch and a few
>> others have been failing for over a month also.
>> I will check in again next month to see how things are progressing
>> Thanks
>> Gav...
>> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>> Hi Gav,
>>>
>>> Looking at the most recent results, I see that the job failed because of
>>> two unit test failures. These are something that will be looked at prior
>>> to the next 1.4.x release which is about to get off the ground.
>>>
>>> I'd kindly request that you not disable the job. Thanks for trying to
>>> find extra resources on these nodes.
>>>
>>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>>>
>>>> can someone take a look into this, the job isnt much good if it is
>>> failing
>>>> all the time and even worse if it is being ignored.
>>>>
>>>> Otherwise I'll disable the job in a dew days to release these wasted
>>>> resources
>>>> to builds that matter.
>>>>
>>>>
>>>
Re: HBase nightly job failing forever
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
I suspect the bigger issue is that the hbase tests are running on the ‘ubuntu’ machines. Since they only have ~300GB for workspaces, the hbase tests are eating a significant majority of it and likely could be dying randomly due to space issues. [All the hbase workspace directories + the yetus-m2 shared mvn cache dirs easily consume 20%+ of the space. Significantly more than the 50 or so other jobs that run on those machines.]
By comparison, most of the ‘Hadoop’ nodes have 2-3TB for the big jobs to consume….
> On Jul 24, 2018, at 8:58 AM, Josh Elser <el...@apache.org> wrote:
>
> Yep, sadly this is a very long tent-pole for us. There are many involved who have invested countless hours in making this better.
>
> Specific to that job you linked earlier, 3 test failures out of our total 4958 tests (0.06% failure rate) is all but "green" in my mind. I would ask that you keep that in mind, too.
>
> To that extent, others have also built another job specifically to find tests which are failing intermittently: https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html. I mention this as evidence to prove to you that this is not a baseless request from the HBase PMC ;)
>
> On 7/24/18 3:14 AM, Gav wrote:
>> Ok, good enough, will wait, please also note 'master' branch and a few
>> others have been failing for over a month also.
>> I will check in again next month to see how things are progressing
>> Thanks
>> Gav...
>> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>>> Hi Gav,
>>>
>>> Looking at the most recent results, I see that the job failed because of
>>> two unit test failures. These are something that will be looked at prior
>>> to the next 1.4.x release which is about to get off the ground.
>>>
>>> I'd kindly request that you not disable the job. Thanks for trying to
>>> find extra resources on these nodes.
>>>
>>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>>>
>>>> can someone take a look into this, the job isnt much good if it is
>>> failing
>>>> all the time and even worse if it is being ignored.
>>>>
>>>> Otherwise I'll disable the job in a dew days to release these wasted
>>>> resources
>>>> to builds that matter.
>>>>
>>>>
>>>
Re: HBase nightly job failing forever
Posted by Josh Elser <el...@apache.org>.
Yep, sadly this is a very long tent-pole for us. There are many involved
who have invested countless hours in making this better.
Specific to that job you linked earlier, 3 test failures out of our
total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
would ask that you keep that in mind, too.
To that extent, others have also built another job specifically to find
tests which are failing intermittently:
https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
I mention this as evidence to prove to you that this is not a baseless
request from the HBase PMC ;)
On 7/24/18 3:14 AM, Gav wrote:
> Ok, good enough, will wait, please also note 'master' branch and a few
> others have been failing for over a month also.
>
> I will check in again next month to see how things are progressing
>
> Thanks
>
> Gav...
>
>
> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>
>> Hi Gav,
>>
>> Looking at the most recent results, I see that the job failed because of
>> two unit test failures. These are something that will be looked at prior
>> to the next 1.4.x release which is about to get off the ground.
>>
>> I'd kindly request that you not disable the job. Thanks for trying to
>> find extra resources on these nodes.
>>
>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>>
>>> can someone take a look into this, the job isnt much good if it is
>> failing
>>> all the time and even worse if it is being ignored.
>>>
>>> Otherwise I'll disable the job in a dew days to release these wasted
>>> resources
>>> to builds that matter.
>>>
>>>
>>
>
>
Re: HBase nightly job failing forever
Posted by Josh Elser <el...@apache.org>.
Yep, sadly this is a very long tent-pole for us. There are many involved
who have invested countless hours in making this better.
Specific to that job you linked earlier, 3 test failures out of our
total 4958 tests (0.06% failure rate) is all but "green" in my mind. I
would ask that you keep that in mind, too.
To that extent, others have also built another job specifically to find
tests which are failing intermittently:
https://builds.apache.org/job/HBase-Find-Flaky-Tests/25513/artifact/dashboard.html.
I mention this as evidence to prove to you that this is not a baseless
request from the HBase PMC ;)
On 7/24/18 3:14 AM, Gav wrote:
> Ok, good enough, will wait, please also note 'master' branch and a few
> others have been failing for over a month also.
>
> I will check in again next month to see how things are progressing
>
> Thanks
>
> Gav...
>
>
> On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
>
>> Hi Gav,
>>
>> Looking at the most recent results, I see that the job failed because of
>> two unit test failures. These are something that will be looked at prior
>> to the next 1.4.x release which is about to get off the ground.
>>
>> I'd kindly request that you not disable the job. Thanks for trying to
>> find extra resources on these nodes.
>>
>> On 7/23/18 12:22 AM, Gavin McDonald wrote:
>>> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>>>
>>> can someone take a look into this, the job isnt much good if it is
>> failing
>>> all the time and even worse if it is being ignored.
>>>
>>> Otherwise I'll disable the job in a dew days to release these wasted
>>> resources
>>> to builds that matter.
>>>
>>>
>>
>
>
Re: HBase nightly job failing forever
Posted by Gav <ip...@gmail.com>.
Ok, good enough, will wait, please also note 'master' branch and a few
others have been failing for over a month also.
I will check in again next month to see how things are progressing
Thanks
Gav...
On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
> Hi Gav,
>
> Looking at the most recent results, I see that the job failed because of
> two unit test failures. These are something that will be looked at prior
> to the next 1.4.x release which is about to get off the ground.
>
> I'd kindly request that you not disable the job. Thanks for trying to
> find extra resources on these nodes.
>
> On 7/23/18 12:22 AM, Gavin McDonald wrote:
> > https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
> >
> > can someone take a look into this, the job isnt much good if it is
> failing
> > all the time and even worse if it is being ignored.
> >
> > Otherwise I'll disable the job in a dew days to release these wasted
> > resources
> > to builds that matter.
> >
> >
>
--
Gav...
Re: HBase nightly job failing forever
Posted by Gav <ip...@gmail.com>.
Ok, good enough, will wait, please also note 'master' branch and a few
others have been failing for over a month also.
I will check in again next month to see how things are progressing
Thanks
Gav...
On Tue, Jul 24, 2018 at 1:19 AM Josh Elser <el...@apache.org> wrote:
> Hi Gav,
>
> Looking at the most recent results, I see that the job failed because of
> two unit test failures. These are something that will be looked at prior
> to the next 1.4.x release which is about to get off the ground.
>
> I'd kindly request that you not disable the job. Thanks for trying to
> find extra resources on these nodes.
>
> On 7/23/18 12:22 AM, Gavin McDonald wrote:
> > https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
> >
> > can someone take a look into this, the job isnt much good if it is
> failing
> > all the time and even worse if it is being ignored.
> >
> > Otherwise I'll disable the job in a dew days to release these wasted
> > resources
> > to builds that matter.
> >
> >
>
--
Gav...
Re: HBase nightly job failing forever
Posted by Josh Elser <el...@apache.org>.
Hi Gav,
Looking at the most recent results, I see that the job failed because of
two unit test failures. These are something that will be looked at prior
to the next 1.4.x release which is about to get off the ground.
I'd kindly request that you not disable the job. Thanks for trying to
find extra resources on these nodes.
On 7/23/18 12:22 AM, Gavin McDonald wrote:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>
> can someone take a look into this, the job isnt much good if it is failing
> all the time and even worse if it is being ignored.
>
> Otherwise I'll disable the job in a dew days to release these wasted
> resources
> to builds that matter.
>
>
Re: HBase nightly job failing forever
Posted by Josh Elser <el...@apache.org>.
Hi Gav,
Looking at the most recent results, I see that the job failed because of
two unit test failures. These are something that will be looked at prior
to the next 1.4.x release which is about to get off the ground.
I'd kindly request that you not disable the job. Thanks for trying to
find extra resources on these nodes.
On 7/23/18 12:22 AM, Gavin McDonald wrote:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/
>
> can someone take a look into this, the job isnt much good if it is failing
> all the time and even worse if it is being ignored.
>
> Otherwise I'll disable the job in a dew days to release these wasted
> resources
> to builds that matter.
>
>