You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Szehon Ho <sz...@cloudera.com> on 2014/05/21 23:09:12 UTC
Re: Plan: permanently move hive builds from bigtop
The pre-commit builds should be officially good now. The test logs are
properly downloaded, and correctly displayed on the HiveQA JIRA comment as
the link under "TestReport", and all logs themselves in raw form are at
"Test logs", see HIVE-6937 and HIVE-7080 for details.
Also as we had troubles with the patch-upload trigger job on Apache Jenkins
having outages and not triggering the test, so we also moved that to the
Hive Jenkins on the same server:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/,
which now runs the hadoopQA script every 10 minutes.
I'll continue to monitor this one, as its still new.
Also as some might know already, the Pre-commit build is now using
-Phadoop-2 and JDK1.7.
Thanks
Szehon
<http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/>
On Tue, Apr 22, 2014 at 4:15 PM, Szehon Ho <sz...@cloudera.com> wrote:
> Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me
> fix the Jenkins url from Apache-Infra side. Now there's no need to
> manually submit Pre-commit builds, unless it got missed somehow. And in
> this Jenkins, you can easily see all the JIRA's in the queue:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/
>
> Sorry, please use this URL and not the first one, as job was renamed due
> to the auto-trigger expecting this format.
>
> And to repeat, Test Reporting and the 'Logs located at:' link is still
> broken and tracked in HIVE-6937. For now, to see the logs of a failed
> build, you have to manually navigate.
> Example, for PreCommit-Hive-BUILD#6, logs are at:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/<http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/>.
> The HiveQA comment on the JIRA should mention the build number.
>
> Thanks,
> Szehon
>
>
> On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
>> It looks great, thanks Lefty!
>>
>>
>> On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz <le...@gmail.com>wrote:
>>
>>> Nice doc, Szehon. I did some minor editing so you might want to make
>>> sure
>>> I didn't introduce any errors.
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
>>>
>>> -- Lefty
>>>
>>>
>>> On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho <sz...@cloudera.com> wrote:
>>>
>>> > Migration is done, I updated the wiki to add all the details of the new
>>> > setup:
>>> >
>>> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
>>> >
>>> > New Jenkins URL to submit pre-commit jobs:
>>> >
>>> >
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/
>>> > Again, this has to be done manually for time being, by clicking on
>>> 'build
>>> > with parameters', and entering the issue number as a parameter. I've
>>> > submitted some already. I'll reach out to some committers to get the
>>> > auto-trigger working.
>>> >
>>> > As I mentioned, there is some work to fix the test-reporting, due to
>>> the
>>> > framework using old url scheme. I am tracking it at
>>> > HIVE-6937<https://issues.apache.org/jira/browse/HIVE-6937>.
>>> > For now I am hosting log directory separately, if you want to see test
>>> > logs, you have to manually go the url corresponding to your build,
>>> like:
>>> >
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for
>>> > run#11. Sorry about that.
>>> >
>>> > Let me know if you see other issues, thanks!
>>> > Szehon
>>> >
>>> >
>>> > On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair <th...@hortonworks.com>
>>> > wrote:
>>> >
>>> > > Sounds good.
>>> > > Thanks Szehon!
>>> > >
>>> > >
>>> > > On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan <
>>> hashutosh@apache.org
>>> > >
>>> > > wrote:
>>> > > > +1 Thanks Szehon!
>>> > > >
>>> > > >
>>> > > > On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang <xz...@cloudera.com>
>>> > > wrote:
>>> > > >
>>> > > >> +1. Thanks for taking care of this.
>>> > > >>
>>> > > >>
>>> > > >> On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho <sz...@cloudera.com>
>>> > > wrote:
>>> > > >>
>>> > > >> > Hi,
>>> > > >> >
>>> > > >> > This week the machine running Hive builds at
>>> > > >> > http://bigtop01.cloudera.org:8080/view/Hive/? ran out of
>>> space, so
>>> > > new
>>> > > >> > jobs like Precommit tests stopped. Its still not resolved
>>> there,
>>> > > there
>>> > > >> was
>>> > > >> > another email today on Bigtop list, but there's very few people
>>> with
>>> > > root
>>> > > >> > access to that host, and they still haven't responded.
>>> > > >> >
>>> > > >> > I chatted with Brock, he has also seen various issues with
>>> Bigtop
>>> > > jenkins
>>> > > >> > in the past, so I am thinking to move the Jenkins jobs to the
>>> PTest
>>> > > >> master
>>> > > >> > itself, where some PMC already have access and can admin if
>>> needed.
>>> > > >> > Currently I am hosting the pre-commit Jenkins job on my own EC2
>>> > > instance
>>> > > >> > as stop-gap.
>>> > > >> >
>>> > > >> > Other advantages of hosting our own Jenkins:
>>> > > >> > 1. No need to wait for other Bigtop jobs to run.
>>> > > >> > 2. Bigtop is using a version of Jenkins that doesnt show
>>> parameters
>>> > > like
>>> > > >> > JIRA number for queued jobs, so impossible to tell whether a
>>> patch
>>> > got
>>> > > >> > picked up and where it is in queue.
>>> > > >> > 3. Eliminate network hop from Bigtop box to our PTest master.
>>> > > >> >
>>> > > >> > The disadvantage is:
>>> > > >> > 1. We don't have much experience doing Jenkins admin, but it
>>> doesn't
>>> > > look
>>> > > >> > too bad. Mostly, restart if there's issue and clean up if out
>>> of
>>> > > space.
>>> > > >> >
>>> > > >> > I wonder what people think, and if there's any objections to
>>> this?
>>> > If
>>> > > >> not,
>>> > > >> > I'll try setting up this weekend. Then, there is some follow-up
>>> > work,
>>> > > >> like
>>> > > >> > changing the Jenkins url's displayed in the test report.
>>> > > >> >
>>> > > >> > Thanks!
>>> > > >> > Szehon
>>> > > >> >
>>> > > >>
>>> > >
>>> > > --
>>> > > CONFIDENTIALITY NOTICE
>>> > > NOTICE: This message is intended for the use of the individual or
>>> entity
>>> > to
>>> > > which it is addressed and may contain information that is
>>> confidential,
>>> > > privileged and exempt from disclosure under applicable law. If the
>>> reader
>>> > > of this message is not the intended recipient, you are hereby
>>> notified
>>> > that
>>> > > any printing, copying, dissemination, distribution, disclosure or
>>> > > forwarding of this communication is strictly prohibited. If you have
>>> > > received this communication in error, please contact the sender
>>> > immediately
>>> > > and delete it from your system. Thank You.
>>> > >
>>> >
>>>
>>
>>
>
Re: Plan: permanently move hive builds from bigtop
Posted by Ashutosh Chauhan <ha...@apache.org>.
Thanks Szehon for your time and contribution in fixing test infra! Looks
like things are good now.
On Wed, May 21, 2014 at 2:09 PM, Szehon Ho <sz...@cloudera.com> wrote:
> The pre-commit builds should be officially good now. The test logs are
> properly downloaded, and correctly displayed on the HiveQA JIRA comment as
> the link under "TestReport", and all logs themselves in raw form are at
> "Test logs", see HIVE-6937 and HIVE-7080 for details.
>
> Also as we had troubles with the patch-upload trigger job on Apache Jenkins
> having outages and not triggering the test, so we also moved that to the
> Hive Jenkins on the same server:
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/
> ,
> which now runs the hadoopQA script every 10 minutes.
> I'll continue to monitor this one, as its still new.
>
> Also as some might know already, the Pre-commit build is now using
> -Phadoop-2 and JDK1.7.
>
> Thanks
> Szehon
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-Admin/
> >
>
>
> On Tue, Apr 22, 2014 at 4:15 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
> > Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me
> > fix the Jenkins url from Apache-Infra side. Now there's no need to
> > manually submit Pre-commit builds, unless it got missed somehow. And in
> > this Jenkins, you can easily see all the JIRA's in the queue:
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/
> >
> > Sorry, please use this URL and not the first one, as job was renamed due
> > to the auto-trigger expecting this format.
> >
> > And to repeat, Test Reporting and the 'Logs located at:' link is still
> > broken and tracked in HIVE-6937. For now, to see the logs of a failed
> > build, you have to manually navigate.
> > Example, for PreCommit-Hive-BUILD#6, logs are at:
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/
> >.
> > The HiveQA comment on the JIRA should mention the build number.
> >
> > Thanks,
> > Szehon
> >
> >
> > On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho <sz...@cloudera.com> wrote:
> >
> >> It looks great, thanks Lefty!
> >>
> >>
> >> On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz <
> leftyleverenz@gmail.com>wrote:
> >>
> >>> Nice doc, Szehon. I did some minor editing so you might want to make
> >>> sure
> >>> I didn't introduce any errors.
> >>>
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
> >>>
> >>> -- Lefty
> >>>
> >>>
> >>> On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho <sz...@cloudera.com>
> wrote:
> >>>
> >>> > Migration is done, I updated the wiki to add all the details of the
> new
> >>> > setup:
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
> >>> >
> >>> > New Jenkins URL to submit pre-commit jobs:
> >>> >
> >>> >
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/
> >>> > Again, this has to be done manually for time being, by clicking on
> >>> 'build
> >>> > with parameters', and entering the issue number as a parameter. I've
> >>> > submitted some already. I'll reach out to some committers to get the
> >>> > auto-trigger working.
> >>> >
> >>> > As I mentioned, there is some work to fix the test-reporting, due to
> >>> the
> >>> > framework using old url scheme. I am tracking it at
> >>> > HIVE-6937<https://issues.apache.org/jira/browse/HIVE-6937>.
> >>> > For now I am hosting log directory separately, if you want to see
> test
> >>> > logs, you have to manually go the url corresponding to your build,
> >>> like:
> >>> >
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for
> >>> > run#11. Sorry about that.
> >>> >
> >>> > Let me know if you see other issues, thanks!
> >>> > Szehon
> >>> >
> >>> >
> >>> > On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair <thejas@hortonworks.com
> >
> >>> > wrote:
> >>> >
> >>> > > Sounds good.
> >>> > > Thanks Szehon!
> >>> > >
> >>> > >
> >>> > > On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan <
> >>> hashutosh@apache.org
> >>> > >
> >>> > > wrote:
> >>> > > > +1 Thanks Szehon!
> >>> > > >
> >>> > > >
> >>> > > > On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang <
> xzhang@cloudera.com>
> >>> > > wrote:
> >>> > > >
> >>> > > >> +1. Thanks for taking care of this.
> >>> > > >>
> >>> > > >>
> >>> > > >> On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho <
> szehon@cloudera.com>
> >>> > > wrote:
> >>> > > >>
> >>> > > >> > Hi,
> >>> > > >> >
> >>> > > >> > This week the machine running Hive builds at
> >>> > > >> > http://bigtop01.cloudera.org:8080/view/Hive/? ran out of
> >>> space, so
> >>> > > new
> >>> > > >> > jobs like Precommit tests stopped. Its still not resolved
> >>> there,
> >>> > > there
> >>> > > >> was
> >>> > > >> > another email today on Bigtop list, but there's very few
> people
> >>> with
> >>> > > root
> >>> > > >> > access to that host, and they still haven't responded.
> >>> > > >> >
> >>> > > >> > I chatted with Brock, he has also seen various issues with
> >>> Bigtop
> >>> > > jenkins
> >>> > > >> > in the past, so I am thinking to move the Jenkins jobs to the
> >>> PTest
> >>> > > >> master
> >>> > > >> > itself, where some PMC already have access and can admin if
> >>> needed.
> >>> > > >> > Currently I am hosting the pre-commit Jenkins job on my own
> EC2
> >>> > > instance
> >>> > > >> > as stop-gap.
> >>> > > >> >
> >>> > > >> > Other advantages of hosting our own Jenkins:
> >>> > > >> > 1. No need to wait for other Bigtop jobs to run.
> >>> > > >> > 2. Bigtop is using a version of Jenkins that doesnt show
> >>> parameters
> >>> > > like
> >>> > > >> > JIRA number for queued jobs, so impossible to tell whether a
> >>> patch
> >>> > got
> >>> > > >> > picked up and where it is in queue.
> >>> > > >> > 3. Eliminate network hop from Bigtop box to our PTest master.
> >>> > > >> >
> >>> > > >> > The disadvantage is:
> >>> > > >> > 1. We don't have much experience doing Jenkins admin, but it
> >>> doesn't
> >>> > > look
> >>> > > >> > too bad. Mostly, restart if there's issue and clean up if out
> >>> of
> >>> > > space.
> >>> > > >> >
> >>> > > >> > I wonder what people think, and if there's any objections to
> >>> this?
> >>> > If
> >>> > > >> not,
> >>> > > >> > I'll try setting up this weekend. Then, there is some
> follow-up
> >>> > work,
> >>> > > >> like
> >>> > > >> > changing the Jenkins url's displayed in the test report.
> >>> > > >> >
> >>> > > >> > Thanks!
> >>> > > >> > Szehon
> >>> > > >> >
> >>> > > >>
> >>> > >
> >>> > > --
> >>> > > CONFIDENTIALITY NOTICE
> >>> > > NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>> > to
> >>> > > which it is addressed and may contain information that is
> >>> confidential,
> >>> > > privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>> > > of this message is not the intended recipient, you are hereby
> >>> notified
> >>> > that
> >>> > > any printing, copying, dissemination, distribution, disclosure or
> >>> > > forwarding of this communication is strictly prohibited. If you
> have
> >>> > > received this communication in error, please contact the sender
> >>> > immediately
> >>> > > and delete it from your system. Thank You.
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>