You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Brock Noland <br...@cloudera.com> on 2013/10/24 21:18:03 UTC
Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Hey guys,
Hive doesn't have any dedicated Apache Jenkins executors and sometimes
our precommit jobs wait for hours to execute. I'd like to move our
jenkins jobs to the BigTop jenkins.
Post move, the Hive project would remain 100% responsible for
maintaining and debugging our jobs. The only thing I see required on
the BigTop front is creating accounts for the hive team members who
need access.
Thoughts?
Cheers,
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Thank you guys so much!
On Tue, Oct 29, 2013 at 12:50 PM, Mark Grover
<gr...@gmail.com>wrote:
> Done!
>
>
> On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
>
> > On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> > >wrote:
> >
> > > > My username is brock.
> > >
> > > I gave you perms to manipulate the jobs. Go wild ;-)
> > >
> >
> > Would you be able to add a "throttle category" for us? Call it
> "hive-unit"
> > perhaps?
> >
> > Thank you!!
> > Brock
> >
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Thank you guys so much!
On Tue, Oct 29, 2013 at 12:50 PM, Mark Grover
<gr...@gmail.com>wrote:
> Done!
>
>
> On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
>
> > On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> > >wrote:
> >
> > > > My username is brock.
> > >
> > > I gave you perms to manipulate the jobs. Go wild ;-)
> > >
> >
> > Would you be able to add a "throttle category" for us? Call it
> "hive-unit"
> > perhaps?
> >
> > Thank you!!
> > Brock
> >
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Mark Grover <gr...@gmail.com>.
Done!
On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
> On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> >wrote:
>
> > > My username is brock.
> >
> > I gave you perms to manipulate the jobs. Go wild ;-)
> >
>
> Would you be able to add a "throttle category" for us? Call it "hive-unit"
> perhaps?
>
> Thank you!!
> Brock
>
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Mark Grover <gr...@gmail.com>.
Done!
On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
> On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> >wrote:
>
> > > My username is brock.
> >
> > I gave you perms to manipulate the jobs. Go wild ;-)
> >
>
> Would you be able to add a "throttle category" for us? Call it "hive-unit"
> perhaps?
>
> Thank you!!
> Brock
>
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> > My username is brock.
>
> I gave you perms to manipulate the jobs. Go wild ;-)
>
Would you be able to add a "throttle category" for us? Call it "hive-unit"
perhaps?
Thank you!!
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, Oct 28, 2013 at 08:51PM, Roman Shaposhnik wrote:
> On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> > Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> > slave is actually just a very lightweight REST client that communicates
> > with our "parallel test" service.
>
> Can you please provide more details on this?
>
> > My username is brock.
>
> I gave you perms to manipulate the jobs. Go wild ;-)
In some definition of "wild", if course ;)
Cos
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> > My username is brock.
>
> I gave you perms to manipulate the jobs. Go wild ;-)
>
Would you be able to add a "throttle category" for us? Call it "hive-unit"
perhaps?
Thank you!!
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> > Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> > slave is actually just a very lightweight REST client that communicates
> > with our "parallel test" service.
>
> Can you please provide more details on this?
>
Our unit tests take like 15 hours if run sequentially so we wrote this
https://github.com/apache/hive/tree/trunk/testutils/ptest2
to execute them in parallel. It executes on a dedicated host and the
jenkins job is just a client of said server.
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> > Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> > slave is actually just a very lightweight REST client that communicates
> > with our "parallel test" service.
>
> Can you please provide more details on this?
>
Our unit tests take like 15 hours if run sequentially so we wrote this
https://github.com/apache/hive/tree/trunk/testutils/ptest2
to execute them in parallel. It executes on a dedicated host and the
jenkins job is just a client of said server.
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> slave is actually just a very lightweight REST client that communicates
> with our "parallel test" service.
Can you please provide more details on this?
> My username is brock.
I gave you perms to manipulate the jobs. Go wild ;-)
Thanks,
Roman.
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> slave is actually just a very lightweight REST client that communicates
> with our "parallel test" service.
Can you please provide more details on this?
> My username is brock.
I gave you perms to manipulate the jobs. Go wild ;-)
Thanks,
Roman.
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Ok sounds good. FWIW, the code that will actually execute on the Jenkins
slave is actually just a very lightweight REST client that communicates
with our "parallel test" service.
My username is brock.
Hive Dev: I think that at least two more committers should create accounts
so we don't have a bus factor here.
Brock
On Mon, Oct 28, 2013 at 1:06 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> Hi Brock,
>
> For as long as these jobs don't block Bigtop
> builds too much -- I'd love to help.
>
> I think the easiest would be for you to register
> on our jenkins:
> http://bigtop01.cloudera.org:8080/
> and then let me know your creds. I can
> give you enough karma to manage jobs/etc.
>
> Given that we already have a few jobs
> running unit tests:
> http://bigtop01.cloudera.org:8080/view/UnitTests/
> you can just follow those examples and
> set up yours.
>
> Thanks,
> Roman.
>
> On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> > This proposal already has support from the Hive community but adding
> > dev@hive as an FYI.
> >
> > On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >> Hey guys,
> >>
> >> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> >> our precommit jobs wait for hours to execute. I'd like to move our
> >> jenkins jobs to the BigTop jenkins.
> >>
> >> Post move, the Hive project would remain 100% responsible for
> >> maintaining and debugging our jobs. The only thing I see required on
> >> the BigTop front is creating accounts for the hive team members who
> >> need access.
> >>
> >> Thoughts?
> >>
> >> Cheers,
> >> Brock
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Ok sounds good. FWIW, the code that will actually execute on the Jenkins
slave is actually just a very lightweight REST client that communicates
with our "parallel test" service.
My username is brock.
Hive Dev: I think that at least two more committers should create accounts
so we don't have a bus factor here.
Brock
On Mon, Oct 28, 2013 at 1:06 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> Hi Brock,
>
> For as long as these jobs don't block Bigtop
> builds too much -- I'd love to help.
>
> I think the easiest would be for you to register
> on our jenkins:
> http://bigtop01.cloudera.org:8080/
> and then let me know your creds. I can
> give you enough karma to manage jobs/etc.
>
> Given that we already have a few jobs
> running unit tests:
> http://bigtop01.cloudera.org:8080/view/UnitTests/
> you can just follow those examples and
> set up yours.
>
> Thanks,
> Roman.
>
> On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> > This proposal already has support from the Hive community but adding
> > dev@hive as an FYI.
> >
> > On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >> Hey guys,
> >>
> >> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> >> our precommit jobs wait for hours to execute. I'd like to move our
> >> jenkins jobs to the BigTop jenkins.
> >>
> >> Post move, the Hive project would remain 100% responsible for
> >> maintaining and debugging our jobs. The only thing I see required on
> >> the BigTop front is creating accounts for the hive team members who
> >> need access.
> >>
> >> Thoughts?
> >>
> >> Cheers,
> >> Brock
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Brock,
For as long as these jobs don't block Bigtop
builds too much -- I'd love to help.
I think the easiest would be for you to register
on our jenkins:
http://bigtop01.cloudera.org:8080/
and then let me know your creds. I can
give you enough karma to manage jobs/etc.
Given that we already have a few jobs
running unit tests:
http://bigtop01.cloudera.org:8080/view/UnitTests/
you can just follow those examples and
set up yours.
Thanks,
Roman.
On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> This proposal already has support from the Hive community but adding
> dev@hive as an FYI.
>
> On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
>> Hey guys,
>>
>> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
>> our precommit jobs wait for hours to execute. I'd like to move our
>> jenkins jobs to the BigTop jenkins.
>>
>> Post move, the Hive project would remain 100% responsible for
>> maintaining and debugging our jobs. The only thing I see required on
>> the BigTop front is creating accounts for the hive team members who
>> need access.
>>
>> Thoughts?
>>
>> Cheers,
>> Brock
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Brock,
For as long as these jobs don't block Bigtop
builds too much -- I'd love to help.
I think the easiest would be for you to register
on our jenkins:
http://bigtop01.cloudera.org:8080/
and then let me know your creds. I can
give you enough karma to manage jobs/etc.
Given that we already have a few jobs
running unit tests:
http://bigtop01.cloudera.org:8080/view/UnitTests/
you can just follow those examples and
set up yours.
Thanks,
Roman.
On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> This proposal already has support from the Hive community but adding
> dev@hive as an FYI.
>
> On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
>> Hey guys,
>>
>> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
>> our precommit jobs wait for hours to execute. I'd like to move our
>> jenkins jobs to the BigTop jenkins.
>>
>> Post move, the Hive project would remain 100% responsible for
>> maintaining and debugging our jobs. The only thing I see required on
>> the BigTop front is creating accounts for the hive team members who
>> need access.
>>
>> Thoughts?
>>
>> Cheers,
>> Brock
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
This proposal already has support from the Hive community but adding
dev@hive as an FYI.
On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
> Hey guys,
>
> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> our precommit jobs wait for hours to execute. I'd like to move our
> jenkins jobs to the BigTop jenkins.
>
> Post move, the Hive project would remain 100% responsible for
> maintaining and debugging our jobs. The only thing I see required on
> the BigTop front is creating accounts for the hive team members who
> need access.
>
> Thoughts?
>
> Cheers,
> Brock
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
This proposal already has support from the Hive community but adding
dev@hive as an FYI.
On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
> Hey guys,
>
> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> our precommit jobs wait for hours to execute. I'd like to move our
> jenkins jobs to the BigTop jenkins.
>
> Post move, the Hive project would remain 100% responsible for
> maintaining and debugging our jobs. The only thing I see required on
> the BigTop front is creating accounts for the hive team members who
> need access.
>
> Thoughts?
>
> Cheers,
> Brock
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org