You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Brock Noland <br...@cloudera.com> on 2013/10/25 17:17:54 UTC
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
This proposal already has support from the Hive community but adding
dev@hive as an FYI.
On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
> Hey guys,
>
> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> our precommit jobs wait for hours to execute. I'd like to move our
> jenkins jobs to the BigTop jenkins.
>
> Post move, the Hive project would remain 100% responsible for
> maintaining and debugging our jobs. The only thing I see required on
> the BigTop front is creating accounts for the hive team members who
> need access.
>
> Thoughts?
>
> Cheers,
> Brock
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Thank you guys so much!
On Tue, Oct 29, 2013 at 12:50 PM, Mark Grover
<gr...@gmail.com>wrote:
> Done!
>
>
> On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
>
> > On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> > >wrote:
> >
> > > > My username is brock.
> > >
> > > I gave you perms to manipulate the jobs. Go wild ;-)
> > >
> >
> > Would you be able to add a "throttle category" for us? Call it
> "hive-unit"
> > perhaps?
> >
> > Thank you!!
> > Brock
> >
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Thank you guys so much!
On Tue, Oct 29, 2013 at 12:50 PM, Mark Grover
<gr...@gmail.com>wrote:
> Done!
>
>
> On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
>
> > On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> > >wrote:
> >
> > > > My username is brock.
> > >
> > > I gave you perms to manipulate the jobs. Go wild ;-)
> > >
> >
> > Would you be able to add a "throttle category" for us? Call it
> "hive-unit"
> > perhaps?
> >
> > Thank you!!
> > Brock
> >
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Mark Grover <gr...@gmail.com>.
Done!
On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
> On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> >wrote:
>
> > > My username is brock.
> >
> > I gave you perms to manipulate the jobs. Go wild ;-)
> >
>
> Would you be able to add a "throttle category" for us? Call it "hive-unit"
> perhaps?
>
> Thank you!!
> Brock
>
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Mark Grover <gr...@gmail.com>.
Done!
On Tue, Oct 29, 2013 at 8:50 AM, Brock Noland <br...@cloudera.com> wrote:
> On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <roman@shaposhnik.org
> >wrote:
>
> > > My username is brock.
> >
> > I gave you perms to manipulate the jobs. Go wild ;-)
> >
>
> Would you be able to add a "throttle category" for us? Call it "hive-unit"
> perhaps?
>
> Thank you!!
> Brock
>
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> > My username is brock.
>
> I gave you perms to manipulate the jobs. Go wild ;-)
>
Would you be able to add a "throttle category" for us? Call it "hive-unit"
perhaps?
Thank you!!
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, Oct 28, 2013 at 08:51PM, Roman Shaposhnik wrote:
> On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> > Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> > slave is actually just a very lightweight REST client that communicates
> > with our "parallel test" service.
>
> Can you please provide more details on this?
>
> > My username is brock.
>
> I gave you perms to manipulate the jobs. Go wild ;-)
In some definition of "wild", if course ;)
Cos
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> > My username is brock.
>
> I gave you perms to manipulate the jobs. Go wild ;-)
>
Would you be able to add a "throttle category" for us? Call it "hive-unit"
perhaps?
Thank you!!
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> > Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> > slave is actually just a very lightweight REST client that communicates
> > with our "parallel test" service.
>
> Can you please provide more details on this?
>
Our unit tests take like 15 hours if run sequentially so we wrote this
https://github.com/apache/hive/tree/trunk/testutils/ptest2
to execute them in parallel. It executes on a dedicated host and the
jenkins job is just a client of said server.
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
On Mon, Oct 28, 2013 at 10:51 PM, Roman Shaposhnik <ro...@shaposhnik.org>wrote:
> On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> > Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> > slave is actually just a very lightweight REST client that communicates
> > with our "parallel test" service.
>
> Can you please provide more details on this?
>
Our unit tests take like 15 hours if run sequentially so we wrote this
https://github.com/apache/hive/tree/trunk/testutils/ptest2
to execute them in parallel. It executes on a dedicated host and the
jenkins job is just a client of said server.
Brock
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> slave is actually just a very lightweight REST client that communicates
> with our "parallel test" service.
Can you please provide more details on this?
> My username is brock.
I gave you perms to manipulate the jobs. Go wild ;-)
Thanks,
Roman.
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Mon, Oct 28, 2013 at 7:34 AM, Brock Noland <br...@cloudera.com> wrote:
> Ok sounds good. FWIW, the code that will actually execute on the Jenkins
> slave is actually just a very lightweight REST client that communicates
> with our "parallel test" service.
Can you please provide more details on this?
> My username is brock.
I gave you perms to manipulate the jobs. Go wild ;-)
Thanks,
Roman.
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Ok sounds good. FWIW, the code that will actually execute on the Jenkins
slave is actually just a very lightweight REST client that communicates
with our "parallel test" service.
My username is brock.
Hive Dev: I think that at least two more committers should create accounts
so we don't have a bus factor here.
Brock
On Mon, Oct 28, 2013 at 1:06 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> Hi Brock,
>
> For as long as these jobs don't block Bigtop
> builds too much -- I'd love to help.
>
> I think the easiest would be for you to register
> on our jenkins:
> http://bigtop01.cloudera.org:8080/
> and then let me know your creds. I can
> give you enough karma to manage jobs/etc.
>
> Given that we already have a few jobs
> running unit tests:
> http://bigtop01.cloudera.org:8080/view/UnitTests/
> you can just follow those examples and
> set up yours.
>
> Thanks,
> Roman.
>
> On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> > This proposal already has support from the Hive community but adding
> > dev@hive as an FYI.
> >
> > On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >> Hey guys,
> >>
> >> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> >> our precommit jobs wait for hours to execute. I'd like to move our
> >> jenkins jobs to the BigTop jenkins.
> >>
> >> Post move, the Hive project would remain 100% responsible for
> >> maintaining and debugging our jobs. The only thing I see required on
> >> the BigTop front is creating accounts for the hive team members who
> >> need access.
> >>
> >> Thoughts?
> >>
> >> Cheers,
> >> Brock
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Brock Noland <br...@cloudera.com>.
Ok sounds good. FWIW, the code that will actually execute on the Jenkins
slave is actually just a very lightweight REST client that communicates
with our "parallel test" service.
My username is brock.
Hive Dev: I think that at least two more committers should create accounts
so we don't have a bus factor here.
Brock
On Mon, Oct 28, 2013 at 1:06 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> Hi Brock,
>
> For as long as these jobs don't block Bigtop
> builds too much -- I'd love to help.
>
> I think the easiest would be for you to register
> on our jenkins:
> http://bigtop01.cloudera.org:8080/
> and then let me know your creds. I can
> give you enough karma to manage jobs/etc.
>
> Given that we already have a few jobs
> running unit tests:
> http://bigtop01.cloudera.org:8080/view/UnitTests/
> you can just follow those examples and
> set up yours.
>
> Thanks,
> Roman.
>
> On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> > This proposal already has support from the Hive community but adding
> > dev@hive as an FYI.
> >
> > On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >> Hey guys,
> >>
> >> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
> >> our precommit jobs wait for hours to execute. I'd like to move our
> >> jenkins jobs to the BigTop jenkins.
> >>
> >> Post move, the Hive project would remain 100% responsible for
> >> maintaining and debugging our jobs. The only thing I see required on
> >> the BigTop front is creating accounts for the hive team members who
> >> need access.
> >>
> >> Thoughts?
> >>
> >> Cheers,
> >> Brock
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Brock,
For as long as these jobs don't block Bigtop
builds too much -- I'd love to help.
I think the easiest would be for you to register
on our jenkins:
http://bigtop01.cloudera.org:8080/
and then let me know your creds. I can
give you enough karma to manage jobs/etc.
Given that we already have a few jobs
running unit tests:
http://bigtop01.cloudera.org:8080/view/UnitTests/
you can just follow those examples and
set up yours.
Thanks,
Roman.
On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> This proposal already has support from the Hive community but adding
> dev@hive as an FYI.
>
> On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
>> Hey guys,
>>
>> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
>> our precommit jobs wait for hours to execute. I'd like to move our
>> jenkins jobs to the BigTop jenkins.
>>
>> Post move, the Hive project would remain 100% responsible for
>> maintaining and debugging our jobs. The only thing I see required on
>> the BigTop front is creating accounts for the hive team members who
>> need access.
>>
>> Thoughts?
>>
>> Cheers,
>> Brock
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: Proposal to move Hive Apache Jenkins jobs to Bigtop Jenkins?
Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Brock,
For as long as these jobs don't block Bigtop
builds too much -- I'd love to help.
I think the easiest would be for you to register
on our jenkins:
http://bigtop01.cloudera.org:8080/
and then let me know your creds. I can
give you enough karma to manage jobs/etc.
Given that we already have a few jobs
running unit tests:
http://bigtop01.cloudera.org:8080/view/UnitTests/
you can just follow those examples and
set up yours.
Thanks,
Roman.
On Fri, Oct 25, 2013 at 8:17 AM, Brock Noland <br...@cloudera.com> wrote:
> This proposal already has support from the Hive community but adding
> dev@hive as an FYI.
>
> On Thu, Oct 24, 2013 at 2:18 PM, Brock Noland <br...@cloudera.com> wrote:
>> Hey guys,
>>
>> Hive doesn't have any dedicated Apache Jenkins executors and sometimes
>> our precommit jobs wait for hours to execute. I'd like to move our
>> jenkins jobs to the BigTop jenkins.
>>
>> Post move, the Hive project would remain 100% responsible for
>> maintaining and debugging our jobs. The only thing I see required on
>> the BigTop front is creating accounts for the hive team members who
>> need access.
>>
>> Thoughts?
>>
>> Cheers,
>> Brock
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org