You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Xuefu Zhang <xz...@cloudera.com> on 2014/03/19 22:56:05 UTC
Pre-commit test hangs
I had to kill the following, which had been running for more than 7 hours.
Let's see if the problem remains.
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
--Xuefu
Re: Pre-commit test hangs
Posted by Szehon Ho <sz...@cloudera.com>.
Done, disabled except pre-commit + hadoop-1.
Thanks
Szehon
On Thu, Mar 20, 2014 at 7:34 PM, Ashutosh Chauhan <ha...@apache.org>wrote:
> Szehon,
> Can you disable other builds? Seems like we didnt make much progress even
> today.
>
> Thanks,
> Ashutosh
>
>
> On Thu, Mar 20, 2014 at 10:54 AM, Szehon Ho <sz...@cloudera.com> wrote:
>
> > Yea it definitely makes sense. That is removing coverage of hadoop-2 and
> > 0.13, but these can be run on demand or on less frequent schedule like
> > weekly. Maybe Harish has comment for 0.13 branch builds.
> >
> > Thanks
> > Szehon
> >
> >
> > On Thu, Mar 20, 2014 at 10:37 AM, Ashutosh Chauhan <hashutosh@apache.org
> > >wrote:
> >
> > > Just to be clear, only run pre-commit builds (which runs patch on trunk
> > on
> > > hadoop-1). Thoughts?
> > >
> > >
> > > On Thu, Mar 20, 2014 at 10:32 AM, Ashutosh Chauhan <
> hashutosh@apache.org
> > > >wrote:
> > >
> > > > I wonder shall we disable all Hive builds except for trunk on
> hadoop-1
> > to
> > > > expedite the queue. Since, thats the primary combination we are using
> > to
> > > > inform committing patches, that will help to move queue faster.
> Without
> > > > patches getting committed (since queue is not moving fast enough)
> other
> > > > builds are not of much use. Once, incoming patch stream slows down a
> > bit,
> > > > we can reeneable more builds. What do other people think?
> > > >
> > > > Thanks,
> > > > Ashutosh
> > > >
> > > >
> > > > On Wed, Mar 19, 2014 at 4:57 PM, Szehon Ho <sz...@cloudera.com>
> > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Thanks for keeping an eye. There's another point worth mentioning,
> > > that a
> > > >> confusing, others had also reported about the same situation. The
> > > >> pre-commit test run can seem to take long because its waiting for
> the
> > > the
> > > >> PTest server to finish the other hive builds (trunk/0.13), which I
> > > >> believed
> > > >> happened here too. See:
> > > >> http://bigtop01.cloudera.org:8080/view/Hive/builds
> > > >>
> > > >> Pre-commit, Trunk, and 0.13 builds run sequentially on the same
> > > >> infrastructure, and throttle is at the server, thus Jenkins reports
> > more
> > > >> time than the actual runtime.
> > > >>
> > > >> After 0.13 is released, that branch build can be disabled, and it
> will
> > > be
> > > >> back up to better capacity with just pre-commit and trunk builds.
> > > >>
> > > >> Thanks
> > > >> Szehon
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Mar 19, 2014 at 3:43 PM, Xuefu Zhang <xz...@cloudera.com>
> > > wrote:
> > > >>
> > > >> > Okay. That could well be. However, test hanging seemingly happened
> > > >> before.
> > > >> > The queue just gets longer when it does happen.
> > > >> >
> > > >> > Thanks,
> > > >> > Xuefu
> > > >> >
> > > >> >
> > > >> > On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <
> > > hashutosh@apache.org
> > > >> > >wrote:
> > > >> >
> > > >> > > It seems like issue is getting in spot instances. Doesn't seem
> > like
> > > >> > > anything specific to patch or test framework. I dont think we
> have
> > > >> much
> > > >> > > choice there rather than let tests to take whatever time they
> > take.
> > > >> So,
> > > >> > > killing a test run won't help much, I guess.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Ashutosh
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <
> xzhang@cloudera.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > I had to kill the following, which had been running for more
> > than
> > > 7
> > > >> > > hours.
> > > >> > > > Let's see if the problem remains.
> > > >> > > >
> > > >> > > >
> > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
> > > >> > > >
> > > >> > > > --Xuefu
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Pre-commit test hangs
Posted by Ashutosh Chauhan <ha...@apache.org>.
Szehon,
Can you disable other builds? Seems like we didnt make much progress even
today.
Thanks,
Ashutosh
On Thu, Mar 20, 2014 at 10:54 AM, Szehon Ho <sz...@cloudera.com> wrote:
> Yea it definitely makes sense. That is removing coverage of hadoop-2 and
> 0.13, but these can be run on demand or on less frequent schedule like
> weekly. Maybe Harish has comment for 0.13 branch builds.
>
> Thanks
> Szehon
>
>
> On Thu, Mar 20, 2014 at 10:37 AM, Ashutosh Chauhan <hashutosh@apache.org
> >wrote:
>
> > Just to be clear, only run pre-commit builds (which runs patch on trunk
> on
> > hadoop-1). Thoughts?
> >
> >
> > On Thu, Mar 20, 2014 at 10:32 AM, Ashutosh Chauhan <hashutosh@apache.org
> > >wrote:
> >
> > > I wonder shall we disable all Hive builds except for trunk on hadoop-1
> to
> > > expedite the queue. Since, thats the primary combination we are using
> to
> > > inform committing patches, that will help to move queue faster. Without
> > > patches getting committed (since queue is not moving fast enough) other
> > > builds are not of much use. Once, incoming patch stream slows down a
> bit,
> > > we can reeneable more builds. What do other people think?
> > >
> > > Thanks,
> > > Ashutosh
> > >
> > >
> > > On Wed, Mar 19, 2014 at 4:57 PM, Szehon Ho <sz...@cloudera.com>
> wrote:
> > >
> > >> Hi,
> > >>
> > >> Thanks for keeping an eye. There's another point worth mentioning,
> > that a
> > >> confusing, others had also reported about the same situation. The
> > >> pre-commit test run can seem to take long because its waiting for the
> > the
> > >> PTest server to finish the other hive builds (trunk/0.13), which I
> > >> believed
> > >> happened here too. See:
> > >> http://bigtop01.cloudera.org:8080/view/Hive/builds
> > >>
> > >> Pre-commit, Trunk, and 0.13 builds run sequentially on the same
> > >> infrastructure, and throttle is at the server, thus Jenkins reports
> more
> > >> time than the actual runtime.
> > >>
> > >> After 0.13 is released, that branch build can be disabled, and it will
> > be
> > >> back up to better capacity with just pre-commit and trunk builds.
> > >>
> > >> Thanks
> > >> Szehon
> > >>
> > >>
> > >>
> > >> On Wed, Mar 19, 2014 at 3:43 PM, Xuefu Zhang <xz...@cloudera.com>
> > wrote:
> > >>
> > >> > Okay. That could well be. However, test hanging seemingly happened
> > >> before.
> > >> > The queue just gets longer when it does happen.
> > >> >
> > >> > Thanks,
> > >> > Xuefu
> > >> >
> > >> >
> > >> > On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <
> > hashutosh@apache.org
> > >> > >wrote:
> > >> >
> > >> > > It seems like issue is getting in spot instances. Doesn't seem
> like
> > >> > > anything specific to patch or test framework. I dont think we have
> > >> much
> > >> > > choice there rather than let tests to take whatever time they
> take.
> > >> So,
> > >> > > killing a test run won't help much, I guess.
> > >> > >
> > >> > > Thanks,
> > >> > > Ashutosh
> > >> > >
> > >> > >
> > >> > > On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xzhang@cloudera.com
> >
> > >> > wrote:
> > >> > >
> > >> > > > I had to kill the following, which had been running for more
> than
> > 7
> > >> > > hours.
> > >> > > > Let's see if the problem remains.
> > >> > > >
> > >> > > >
> http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
> > >> > > >
> > >> > > > --Xuefu
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Pre-commit test hangs
Posted by Szehon Ho <sz...@cloudera.com>.
Yea it definitely makes sense. That is removing coverage of hadoop-2 and
0.13, but these can be run on demand or on less frequent schedule like
weekly. Maybe Harish has comment for 0.13 branch builds.
Thanks
Szehon
On Thu, Mar 20, 2014 at 10:37 AM, Ashutosh Chauhan <ha...@apache.org>wrote:
> Just to be clear, only run pre-commit builds (which runs patch on trunk on
> hadoop-1). Thoughts?
>
>
> On Thu, Mar 20, 2014 at 10:32 AM, Ashutosh Chauhan <hashutosh@apache.org
> >wrote:
>
> > I wonder shall we disable all Hive builds except for trunk on hadoop-1 to
> > expedite the queue. Since, thats the primary combination we are using to
> > inform committing patches, that will help to move queue faster. Without
> > patches getting committed (since queue is not moving fast enough) other
> > builds are not of much use. Once, incoming patch stream slows down a bit,
> > we can reeneable more builds. What do other people think?
> >
> > Thanks,
> > Ashutosh
> >
> >
> > On Wed, Mar 19, 2014 at 4:57 PM, Szehon Ho <sz...@cloudera.com> wrote:
> >
> >> Hi,
> >>
> >> Thanks for keeping an eye. There's another point worth mentioning,
> that a
> >> confusing, others had also reported about the same situation. The
> >> pre-commit test run can seem to take long because its waiting for the
> the
> >> PTest server to finish the other hive builds (trunk/0.13), which I
> >> believed
> >> happened here too. See:
> >> http://bigtop01.cloudera.org:8080/view/Hive/builds
> >>
> >> Pre-commit, Trunk, and 0.13 builds run sequentially on the same
> >> infrastructure, and throttle is at the server, thus Jenkins reports more
> >> time than the actual runtime.
> >>
> >> After 0.13 is released, that branch build can be disabled, and it will
> be
> >> back up to better capacity with just pre-commit and trunk builds.
> >>
> >> Thanks
> >> Szehon
> >>
> >>
> >>
> >> On Wed, Mar 19, 2014 at 3:43 PM, Xuefu Zhang <xz...@cloudera.com>
> wrote:
> >>
> >> > Okay. That could well be. However, test hanging seemingly happened
> >> before.
> >> > The queue just gets longer when it does happen.
> >> >
> >> > Thanks,
> >> > Xuefu
> >> >
> >> >
> >> > On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <
> hashutosh@apache.org
> >> > >wrote:
> >> >
> >> > > It seems like issue is getting in spot instances. Doesn't seem like
> >> > > anything specific to patch or test framework. I dont think we have
> >> much
> >> > > choice there rather than let tests to take whatever time they take.
> >> So,
> >> > > killing a test run won't help much, I guess.
> >> > >
> >> > > Thanks,
> >> > > Ashutosh
> >> > >
> >> > >
> >> > > On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xz...@cloudera.com>
> >> > wrote:
> >> > >
> >> > > > I had to kill the following, which had been running for more than
> 7
> >> > > hours.
> >> > > > Let's see if the problem remains.
> >> > > >
> >> > > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
> >> > > >
> >> > > > --Xuefu
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Pre-commit test hangs
Posted by Ashutosh Chauhan <ha...@apache.org>.
Just to be clear, only run pre-commit builds (which runs patch on trunk on
hadoop-1). Thoughts?
On Thu, Mar 20, 2014 at 10:32 AM, Ashutosh Chauhan <ha...@apache.org>wrote:
> I wonder shall we disable all Hive builds except for trunk on hadoop-1 to
> expedite the queue. Since, thats the primary combination we are using to
> inform committing patches, that will help to move queue faster. Without
> patches getting committed (since queue is not moving fast enough) other
> builds are not of much use. Once, incoming patch stream slows down a bit,
> we can reeneable more builds. What do other people think?
>
> Thanks,
> Ashutosh
>
>
> On Wed, Mar 19, 2014 at 4:57 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
>> Hi,
>>
>> Thanks for keeping an eye. There's another point worth mentioning, that a
>> confusing, others had also reported about the same situation. The
>> pre-commit test run can seem to take long because its waiting for the the
>> PTest server to finish the other hive builds (trunk/0.13), which I
>> believed
>> happened here too. See:
>> http://bigtop01.cloudera.org:8080/view/Hive/builds
>>
>> Pre-commit, Trunk, and 0.13 builds run sequentially on the same
>> infrastructure, and throttle is at the server, thus Jenkins reports more
>> time than the actual runtime.
>>
>> After 0.13 is released, that branch build can be disabled, and it will be
>> back up to better capacity with just pre-commit and trunk builds.
>>
>> Thanks
>> Szehon
>>
>>
>>
>> On Wed, Mar 19, 2014 at 3:43 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
>>
>> > Okay. That could well be. However, test hanging seemingly happened
>> before.
>> > The queue just gets longer when it does happen.
>> >
>> > Thanks,
>> > Xuefu
>> >
>> >
>> > On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <hashutosh@apache.org
>> > >wrote:
>> >
>> > > It seems like issue is getting in spot instances. Doesn't seem like
>> > > anything specific to patch or test framework. I dont think we have
>> much
>> > > choice there rather than let tests to take whatever time they take.
>> So,
>> > > killing a test run won't help much, I guess.
>> > >
>> > > Thanks,
>> > > Ashutosh
>> > >
>> > >
>> > > On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xz...@cloudera.com>
>> > wrote:
>> > >
>> > > > I had to kill the following, which had been running for more than 7
>> > > hours.
>> > > > Let's see if the problem remains.
>> > > >
>> > > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
>> > > >
>> > > > --Xuefu
>> > > >
>> > >
>> >
>>
>
>
Re: Pre-commit test hangs
Posted by Ashutosh Chauhan <ha...@apache.org>.
I wonder shall we disable all Hive builds except for trunk on hadoop-1 to
expedite the queue. Since, thats the primary combination we are using to
inform committing patches, that will help to move queue faster. Without
patches getting committed (since queue is not moving fast enough) other
builds are not of much use. Once, incoming patch stream slows down a bit,
we can reeneable more builds. What do other people think?
Thanks,
Ashutosh
On Wed, Mar 19, 2014 at 4:57 PM, Szehon Ho <sz...@cloudera.com> wrote:
> Hi,
>
> Thanks for keeping an eye. There's another point worth mentioning, that a
> confusing, others had also reported about the same situation. The
> pre-commit test run can seem to take long because its waiting for the the
> PTest server to finish the other hive builds (trunk/0.13), which I believed
> happened here too. See:
> http://bigtop01.cloudera.org:8080/view/Hive/builds
>
> Pre-commit, Trunk, and 0.13 builds run sequentially on the same
> infrastructure, and throttle is at the server, thus Jenkins reports more
> time than the actual runtime.
>
> After 0.13 is released, that branch build can be disabled, and it will be
> back up to better capacity with just pre-commit and trunk builds.
>
> Thanks
> Szehon
>
>
>
> On Wed, Mar 19, 2014 at 3:43 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
>
> > Okay. That could well be. However, test hanging seemingly happened
> before.
> > The queue just gets longer when it does happen.
> >
> > Thanks,
> > Xuefu
> >
> >
> > On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <hashutosh@apache.org
> > >wrote:
> >
> > > It seems like issue is getting in spot instances. Doesn't seem like
> > > anything specific to patch or test framework. I dont think we have much
> > > choice there rather than let tests to take whatever time they take. So,
> > > killing a test run won't help much, I guess.
> > >
> > > Thanks,
> > > Ashutosh
> > >
> > >
> > > On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xz...@cloudera.com>
> > wrote:
> > >
> > > > I had to kill the following, which had been running for more than 7
> > > hours.
> > > > Let's see if the problem remains.
> > > >
> > > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
> > > >
> > > > --Xuefu
> > > >
> > >
> >
>
Re: Pre-commit test hangs
Posted by Szehon Ho <sz...@cloudera.com>.
Hi,
Thanks for keeping an eye. There's another point worth mentioning, that a
confusing, others had also reported about the same situation. The
pre-commit test run can seem to take long because its waiting for the the
PTest server to finish the other hive builds (trunk/0.13), which I believed
happened here too. See: http://bigtop01.cloudera.org:8080/view/Hive/builds
Pre-commit, Trunk, and 0.13 builds run sequentially on the same
infrastructure, and throttle is at the server, thus Jenkins reports more
time than the actual runtime.
After 0.13 is released, that branch build can be disabled, and it will be
back up to better capacity with just pre-commit and trunk builds.
Thanks
Szehon
On Wed, Mar 19, 2014 at 3:43 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> Okay. That could well be. However, test hanging seemingly happened before.
> The queue just gets longer when it does happen.
>
> Thanks,
> Xuefu
>
>
> On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <hashutosh@apache.org
> >wrote:
>
> > It seems like issue is getting in spot instances. Doesn't seem like
> > anything specific to patch or test framework. I dont think we have much
> > choice there rather than let tests to take whatever time they take. So,
> > killing a test run won't help much, I guess.
> >
> > Thanks,
> > Ashutosh
> >
> >
> > On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xz...@cloudera.com>
> wrote:
> >
> > > I had to kill the following, which had been running for more than 7
> > hours.
> > > Let's see if the problem remains.
> > >
> > > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
> > >
> > > --Xuefu
> > >
> >
>
Re: Pre-commit test hangs
Posted by Xuefu Zhang <xz...@cloudera.com>.
Okay. That could well be. However, test hanging seemingly happened before.
The queue just gets longer when it does happen.
Thanks,
Xuefu
On Wed, Mar 19, 2014 at 3:40 PM, Ashutosh Chauhan <ha...@apache.org>wrote:
> It seems like issue is getting in spot instances. Doesn't seem like
> anything specific to patch or test framework. I dont think we have much
> choice there rather than let tests to take whatever time they take. So,
> killing a test run won't help much, I guess.
>
> Thanks,
> Ashutosh
>
>
> On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
>
> > I had to kill the following, which had been running for more than 7
> hours.
> > Let's see if the problem remains.
> >
> > http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
> >
> > --Xuefu
> >
>
Re: Pre-commit test hangs
Posted by Ashutosh Chauhan <ha...@apache.org>.
It seems like issue is getting in spot instances. Doesn't seem like
anything specific to patch or test framework. I dont think we have much
choice there rather than let tests to take whatever time they take. So,
killing a test run won't help much, I guess.
Thanks,
Ashutosh
On Wed, Mar 19, 2014 at 2:56 PM, Xuefu Zhang <xz...@cloudera.com> wrote:
> I had to kill the following, which had been running for more than 7 hours.
> Let's see if the problem remains.
>
> http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1872/
>
> --Xuefu
>