You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Alex Newman <po...@gmail.com> on 2014/08/27 22:28:49 UTC
I wonder if it makes sense to restrict the HBase build to one build
per jenkins host
I have noticed that HBase tests can cause a good deal of inherent test
interference. We have had much better luck running one and only one
set of the tests at a time. I notice that this currently is not
happening. I don't have any rights on jenkins, so perhaps someone else
could give this a shot. I'd be glad to describe how do it. Another
option would be to run the build from within a container (We do this
at WanDISCO). Finally I got the build running on travis ci. Checkout
https://github.com/OhmData/hbase-public/tree/travis for an example of
how to get this done.
Re: I wonder if it makes sense to restrict the HBase build to one
build per jenkins host
Posted by Stack <st...@duboce.net>.
I filed https://issues.apache.org/jira/browse/BUILDS-15
On Fri, Aug 29, 2014 at 4:13 PM, Stack <st...@duboce.net> wrote:
> Giri:
>
> Any chance of a heads up when you change the jenkins environment. Only
> for Alex digging, we'd not have figured that the new fails were because we
> were running alongside another process. We should make it so hbase tests
> are immune to concurrent test runs for sure we didn't know this was the
> prob.
>
> Via our Esteban, this seems to be showing we are running 4 executors
> https://builds.apache.org/computer/H11/load-statistics Is that so?
>
> Let me file a builds issue with the above question to raise the profile.
> Thanks,
> St.Ack
>
>
> On Wed, Aug 27, 2014 at 6:31 PM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
>> I recently added more than one executors per slave to get past the build
>> queueing along. Let me bring it down to one executors for now if thats
>> causing issues.
>>
>>
>> BTW, there is also a discussion about using dockers for build slaves.
>>
>>
>>
>> -giri
>>
>>
>> On Wed, Aug 27, 2014 at 6:06 PM, Enis Söztutar <en...@apache.org> wrote:
>>
>> > Yes,
>> >
>> > Giri still does some maintenance on those. He was saying that we will
>> > receive some more nodes from rackspace. Let me add him to the thread.
>> >
>> > Enis
>> >
>> >
>> > On Wed, Aug 27, 2014 at 1:59 PM, Nicolas Liochon <nk...@gmail.com>
>> > wrote:
>> >
>> >> It used to be the case for precommit builds: there was a single worker.
>> >> It's much simpler this way.
>> >> It seems it has changed, likely less than a year ago. I see on
>> >> https://builds.apache.org/computer/H0/load-statistics that there are 2
>> >> workers.
>> >> We're can't do what we want on these machines, as they are used by
>> other
>> >> projects however. In the past, Giri was administrating these machines,
>> I
>> >> don't know if it's still the case (Enis? Devaraj? Stack? Do you guys
>> >> know?)
>> >>
>> >> I guess that the root issue is that hdfs & other builds are not
>> >> parallelized as HBase are, so they need multiple workers to really use
>> the
>> >> multiple cores of the build env...
>> >>
>> >> Nicolas
>> >>
>> >>
>> >> On Wed, Aug 27, 2014 at 10:45 PM, Andrew Purtell <ap...@apache.org>
>> >> wrote:
>> >>
>> >> > +1, if it's possible on ASF Jenkins
>> >> >
>> >> > We also share with other projects. I've seen build reports
>> implicating
>> >> > Oozie tests for supposed HBase test zombies.
>> >> >
>> >> > When we had cloud based Jenkins running at Trend Micro and Intel we
>> >> > launched instances on demand as we needed workers, but never more
>> than
>> >> one
>> >> > worker per instance concurrently. Definitely helps to reduce failures
>> >> due
>> >> > to unexpected state from interactions between tests.
>> >> >
>> >> >
>> >> > On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman <po...@gmail.com>
>> wrote:
>> >> >
>> >> > > I have noticed that HBase tests can cause a good deal of inherent
>> test
>> >> > > interference. We have had much better luck running one and only one
>> >> > > set of the tests at a time. I notice that this currently is not
>> >> > > happening. I don't have any rights on jenkins, so perhaps someone
>> else
>> >> > > could give this a shot. I'd be glad to describe how do it. Another
>> >> > > option would be to run the build from within a container (We do
>> this
>> >> > > at WanDISCO). Finally I got the build running on travis ci.
>> Checkout
>> >> > > https://github.com/OhmData/hbase-public/tree/travis for an
>> example of
>> >> > > how to get this done.
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best regards,
>> >> >
>> >> > - Andy
>> >> >
>> >> > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> >> > (via Tom White)
>> >> >
>> >>
>> >
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: I wonder if it makes sense to restrict the HBase build to one
build per jenkins host
Posted by Stack <st...@duboce.net>.
Giri:
Any chance of a heads up when you change the jenkins environment. Only for
Alex digging, we'd not have figured that the new fails were because we were
running alongside another process. We should make it so hbase tests are
immune to concurrent test runs for sure we didn't know this was the prob.
Via our Esteban, this seems to be showing we are running 4 executors
https://builds.apache.org/computer/H11/load-statistics Is that so?
Let me file a builds issue with the above question to raise the profile.
Thanks,
St.Ack
On Wed, Aug 27, 2014 at 6:31 PM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> I recently added more than one executors per slave to get past the build
> queueing along. Let me bring it down to one executors for now if thats
> causing issues.
>
>
> BTW, there is also a discussion about using dockers for build slaves.
>
>
>
> -giri
>
>
> On Wed, Aug 27, 2014 at 6:06 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Yes,
> >
> > Giri still does some maintenance on those. He was saying that we will
> > receive some more nodes from rackspace. Let me add him to the thread.
> >
> > Enis
> >
> >
> > On Wed, Aug 27, 2014 at 1:59 PM, Nicolas Liochon <nk...@gmail.com>
> > wrote:
> >
> >> It used to be the case for precommit builds: there was a single worker.
> >> It's much simpler this way.
> >> It seems it has changed, likely less than a year ago. I see on
> >> https://builds.apache.org/computer/H0/load-statistics that there are 2
> >> workers.
> >> We're can't do what we want on these machines, as they are used by other
> >> projects however. In the past, Giri was administrating these machines, I
> >> don't know if it's still the case (Enis? Devaraj? Stack? Do you guys
> >> know?)
> >>
> >> I guess that the root issue is that hdfs & other builds are not
> >> parallelized as HBase are, so they need multiple workers to really use
> the
> >> multiple cores of the build env...
> >>
> >> Nicolas
> >>
> >>
> >> On Wed, Aug 27, 2014 at 10:45 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >> > +1, if it's possible on ASF Jenkins
> >> >
> >> > We also share with other projects. I've seen build reports implicating
> >> > Oozie tests for supposed HBase test zombies.
> >> >
> >> > When we had cloud based Jenkins running at Trend Micro and Intel we
> >> > launched instances on demand as we needed workers, but never more than
> >> one
> >> > worker per instance concurrently. Definitely helps to reduce failures
> >> due
> >> > to unexpected state from interactions between tests.
> >> >
> >> >
> >> > On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman <po...@gmail.com>
> wrote:
> >> >
> >> > > I have noticed that HBase tests can cause a good deal of inherent
> test
> >> > > interference. We have had much better luck running one and only one
> >> > > set of the tests at a time. I notice that this currently is not
> >> > > happening. I don't have any rights on jenkins, so perhaps someone
> else
> >> > > could give this a shot. I'd be glad to describe how do it. Another
> >> > > option would be to run the build from within a container (We do this
> >> > > at WanDISCO). Finally I got the build running on travis ci.
> Checkout
> >> > > https://github.com/OhmData/hbase-public/tree/travis for an example
> of
> >> > > how to get this done.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >> > (via Tom White)
> >> >
> >>
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: I wonder if it makes sense to restrict the HBase build to one
build per jenkins host
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
I recently added more than one executors per slave to get past the build
queueing along. Let me bring it down to one executors for now if thats
causing issues.
BTW, there is also a discussion about using dockers for build slaves.
-giri
On Wed, Aug 27, 2014 at 6:06 PM, Enis Söztutar <en...@apache.org> wrote:
> Yes,
>
> Giri still does some maintenance on those. He was saying that we will
> receive some more nodes from rackspace. Let me add him to the thread.
>
> Enis
>
>
> On Wed, Aug 27, 2014 at 1:59 PM, Nicolas Liochon <nk...@gmail.com>
> wrote:
>
>> It used to be the case for precommit builds: there was a single worker.
>> It's much simpler this way.
>> It seems it has changed, likely less than a year ago. I see on
>> https://builds.apache.org/computer/H0/load-statistics that there are 2
>> workers.
>> We're can't do what we want on these machines, as they are used by other
>> projects however. In the past, Giri was administrating these machines, I
>> don't know if it's still the case (Enis? Devaraj? Stack? Do you guys
>> know?)
>>
>> I guess that the root issue is that hdfs & other builds are not
>> parallelized as HBase are, so they need multiple workers to really use the
>> multiple cores of the build env...
>>
>> Nicolas
>>
>>
>> On Wed, Aug 27, 2014 at 10:45 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>> > +1, if it's possible on ASF Jenkins
>> >
>> > We also share with other projects. I've seen build reports implicating
>> > Oozie tests for supposed HBase test zombies.
>> >
>> > When we had cloud based Jenkins running at Trend Micro and Intel we
>> > launched instances on demand as we needed workers, but never more than
>> one
>> > worker per instance concurrently. Definitely helps to reduce failures
>> due
>> > to unexpected state from interactions between tests.
>> >
>> >
>> > On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman <po...@gmail.com> wrote:
>> >
>> > > I have noticed that HBase tests can cause a good deal of inherent test
>> > > interference. We have had much better luck running one and only one
>> > > set of the tests at a time. I notice that this currently is not
>> > > happening. I don't have any rights on jenkins, so perhaps someone else
>> > > could give this a shot. I'd be glad to describe how do it. Another
>> > > option would be to run the build from within a container (We do this
>> > > at WanDISCO). Finally I got the build running on travis ci. Checkout
>> > > https://github.com/OhmData/hbase-public/tree/travis for an example of
>> > > how to get this done.
>> > >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>> >
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: I wonder if it makes sense to restrict the HBase build to one
build per jenkins host
Posted by Enis Söztutar <en...@apache.org>.
Yes,
Giri still does some maintenance on those. He was saying that we will
receive some more nodes from rackspace. Let me add him to the thread.
Enis
On Wed, Aug 27, 2014 at 1:59 PM, Nicolas Liochon <nk...@gmail.com> wrote:
> It used to be the case for precommit builds: there was a single worker.
> It's much simpler this way.
> It seems it has changed, likely less than a year ago. I see on
> https://builds.apache.org/computer/H0/load-statistics that there are 2
> workers.
> We're can't do what we want on these machines, as they are used by other
> projects however. In the past, Giri was administrating these machines, I
> don't know if it's still the case (Enis? Devaraj? Stack? Do you guys know?)
>
> I guess that the root issue is that hdfs & other builds are not
> parallelized as HBase are, so they need multiple workers to really use the
> multiple cores of the build env...
>
> Nicolas
>
>
> On Wed, Aug 27, 2014 at 10:45 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > +1, if it's possible on ASF Jenkins
> >
> > We also share with other projects. I've seen build reports implicating
> > Oozie tests for supposed HBase test zombies.
> >
> > When we had cloud based Jenkins running at Trend Micro and Intel we
> > launched instances on demand as we needed workers, but never more than
> one
> > worker per instance concurrently. Definitely helps to reduce failures due
> > to unexpected state from interactions between tests.
> >
> >
> > On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman <po...@gmail.com> wrote:
> >
> > > I have noticed that HBase tests can cause a good deal of inherent test
> > > interference. We have had much better luck running one and only one
> > > set of the tests at a time. I notice that this currently is not
> > > happening. I don't have any rights on jenkins, so perhaps someone else
> > > could give this a shot. I'd be glad to describe how do it. Another
> > > option would be to run the build from within a container (We do this
> > > at WanDISCO). Finally I got the build running on travis ci. Checkout
> > > https://github.com/OhmData/hbase-public/tree/travis for an example of
> > > how to get this done.
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: I wonder if it makes sense to restrict the HBase build to one
build per jenkins host
Posted by Nicolas Liochon <nk...@gmail.com>.
It used to be the case for precommit builds: there was a single worker.
It's much simpler this way.
It seems it has changed, likely less than a year ago. I see on
https://builds.apache.org/computer/H0/load-statistics that there are 2
workers.
We're can't do what we want on these machines, as they are used by other
projects however. In the past, Giri was administrating these machines, I
don't know if it's still the case (Enis? Devaraj? Stack? Do you guys know?)
I guess that the root issue is that hdfs & other builds are not
parallelized as HBase are, so they need multiple workers to really use the
multiple cores of the build env...
Nicolas
On Wed, Aug 27, 2014 at 10:45 PM, Andrew Purtell <ap...@apache.org>
wrote:
> +1, if it's possible on ASF Jenkins
>
> We also share with other projects. I've seen build reports implicating
> Oozie tests for supposed HBase test zombies.
>
> When we had cloud based Jenkins running at Trend Micro and Intel we
> launched instances on demand as we needed workers, but never more than one
> worker per instance concurrently. Definitely helps to reduce failures due
> to unexpected state from interactions between tests.
>
>
> On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman <po...@gmail.com> wrote:
>
> > I have noticed that HBase tests can cause a good deal of inherent test
> > interference. We have had much better luck running one and only one
> > set of the tests at a time. I notice that this currently is not
> > happening. I don't have any rights on jenkins, so perhaps someone else
> > could give this a shot. I'd be glad to describe how do it. Another
> > option would be to run the build from within a container (We do this
> > at WanDISCO). Finally I got the build running on travis ci. Checkout
> > https://github.com/OhmData/hbase-public/tree/travis for an example of
> > how to get this done.
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: I wonder if it makes sense to restrict the HBase build to one
build per jenkins host
Posted by Andrew Purtell <ap...@apache.org>.
+1, if it's possible on ASF Jenkins
We also share with other projects. I've seen build reports implicating
Oozie tests for supposed HBase test zombies.
When we had cloud based Jenkins running at Trend Micro and Intel we
launched instances on demand as we needed workers, but never more than one
worker per instance concurrently. Definitely helps to reduce failures due
to unexpected state from interactions between tests.
On Wed, Aug 27, 2014 at 1:28 PM, Alex Newman <po...@gmail.com> wrote:
> I have noticed that HBase tests can cause a good deal of inherent test
> interference. We have had much better luck running one and only one
> set of the tests at a time. I notice that this currently is not
> happening. I don't have any rights on jenkins, so perhaps someone else
> could give this a shot. I'd be glad to describe how do it. Another
> option would be to run the build from within a container (We do this
> at WanDISCO). Finally I got the build running on travis ci. Checkout
> https://github.com/OhmData/hbase-public/tree/travis for an example of
> how to get this done.
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)