You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Prasanth J <j....@gmail.com> on 2015/12/01 06:09:41 UTC
Precommit unstable?
Hi All
I am noticing something weird in precommit queue. I can see HIVE-12541currenly running, but is also waiting next in the queue. The same jira ran few more times recently. I cannot see any other patches in the queue. Sometime I see bunch of patches in the queue but it will soon be removed. But I am seeing HIVE-12541 running quite often. Has anyone seen this issue?
Thanks
Prasanth
Re: Precommit unstable?
Posted by Thejas Nair <th...@gmail.com>.
fyi, several patches in queue failed to get tested because of some
failures in cloning from github. That could be an unrelated temporary
problem.
On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair <th...@gmail.com> wrote:
> The build running right now has been running for almost 10 hours.
> I am wondering if there is something wrong with the hosts being used
> right now. Some commands seem to be taking longer and I get some
> intermittent errors from the jenkins UI.
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
> This rsync command seems to have taken a long time to finish -
>
> 2015-12-01 11:42:41,394 INFO LocalCommand.<init>:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
> 2015-12-01 12:54:52,522 INFO LocalCommand.<init>:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
> 2015-12-01 12:54:54,240 INFO HostExecutor.executeTests:176 Starting
> isolated execution on 54.157.22.221
>
>
>
>
>
> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena <se...@cloudera.com> wrote:
>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> Available' status was set. I does not say at what time, but Jenkins runs a
>> job every 5 min to detect which files must be tested for all "Patch
>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
>> times.
>>
>> I would recommend to set the JIRA to 'Patch Available' when the patch is
>> reviewed to avoid many patches to be added to the queue.
>>
>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> I am noticing something weird in precommit queue. I can see
>>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>>> jira ran few more times recently. I cannot see any other patches in the
>>> queue. Sometime I see bunch of patches in the queue but it will soon be
>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>>> this issue?
>>>
>>> Thanks
>>> Prasanth
Re: Precommit unstable?
Posted by Thejas Nair <th...@gmail.com>.
Forgot to get back on this one. That delay was actually caused by a
hanging test, it was addressed here -
https://issues.apache.org/jira/browse/HIVE-12560
On Wed, Dec 2, 2015 at 10:08 AM, Sergio Pena <se...@cloudera.com> wrote:
> I see the last tests run in average of 1 hour. Should I still restart
> Jenkins?
>
> On Tue, Dec 1, 2015 at 12:11 PM, Thejas Nair <th...@gmail.com> wrote:
>>
>> I killed the current 10+hr test run to see if next test would run
>> faster. (Discussed with Prasanth, the author of patch being tested).
>>
>> However, I doubt if the next one is going to perform any better. If
>> we could try restarting the ptest2 system, I think that might be
>> better thing to do (instead of waiting several hours to realize that
>> the issue has not gone away).
>>
>>
>> On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair <th...@gmail.com>
>> wrote:
>> > Would restart of jenkins or the hosts help with this ?
>> >
>> >
>> > On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair <th...@gmail.com>
>> > wrote:
>> >> The build running right now has been running for almost 10 hours.
>> >> I am wondering if there is something wrong with the hosts being used
>> >> right now. Some commands seem to be taking longer and I get some
>> >> intermittent errors from the jenkins UI.
>> >>
>> >>
>> >> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
>> >> This rsync command seems to have taken a long time to finish -
>> >>
>> >> 2015-12-01 11:42:41,394 INFO LocalCommand.<init>:35 Starting timeout
>> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> >> --timeout 600
>> >> hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
>> >>
>> >> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
>> >> 2015-12-01 12:54:52,522 INFO LocalCommand.<init>:35 Starting timeout
>> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> >> --timeout 600
>> >> hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
>> >>
>> >> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
>> >> 2015-12-01 12:54:54,240 INFO HostExecutor.executeTests:176 Starting
>> >> isolated execution on 54.157.22.221
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena <se...@cloudera.com>
>> >> wrote:
>> >>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> >>> Available' status was set. I does not say at what time, but Jenkins
>> >>> runs a
>> >>> job every 5 min to detect which files must be tested for all "Patch
>> >>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue
>> >>> many
>> >>> times.
>> >>>
>> >>> I would recommend to set the JIRA to 'Patch Available' when the patch
>> >>> is
>> >>> reviewed to avoid many patches to be added to the queue.
>> >>>
>> >>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi All
>> >>>>
>> >>>> I am noticing something weird in precommit queue. I can see
>> >>>> HIVE-12541currenly running, but is also waiting next in the queue.
>> >>>> The same
>> >>>> jira ran few more times recently. I cannot see any other patches in
>> >>>> the
>> >>>> queue. Sometime I see bunch of patches in the queue but it will soon
>> >>>> be
>> >>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone
>> >>>> seen
>> >>>> this issue?
>> >>>>
>> >>>> Thanks
>> >>>> Prasanth
>
>
Re: Precommit unstable?
Posted by Sergio Pena <se...@cloudera.com>.
I see the last tests run in average of 1 hour. Should I still restart
Jenkins?
On Tue, Dec 1, 2015 at 12:11 PM, Thejas Nair <th...@gmail.com> wrote:
> I killed the current 10+hr test run to see if next test would run
> faster. (Discussed with Prasanth, the author of patch being tested).
>
> However, I doubt if the next one is going to perform any better. If
> we could try restarting the ptest2 system, I think that might be
> better thing to do (instead of waiting several hours to realize that
> the issue has not gone away).
>
>
> On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair <th...@gmail.com>
> wrote:
> > Would restart of jenkins or the hosts help with this ?
> >
> >
> > On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair <th...@gmail.com>
> wrote:
> >> The build running right now has been running for almost 10 hours.
> >> I am wondering if there is something wrong with the hosts being used
> >> right now. Some commands seem to be taking longer and I get some
> >> intermittent errors from the jenkins UI.
> >>
> >>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
> >> This rsync command seems to have taken a long time to finish -
> >>
> >> 2015-12-01 11:42:41,394 INFO LocalCommand.<init>:35 Starting timeout
> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> >> --timeout 600 hiveptest@54.159.202.234:
> /home/hiveptest/54.159.202.234-hiveptest-0/logs/
> >>
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
> >> 2015-12-01 12:54:52,522 INFO LocalCommand.<init>:35 Starting timeout
> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> >> --timeout 600 hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221
> -hiveptest-0/logs/
> >>
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
> >> 2015-12-01 12:54:54,240 INFO HostExecutor.executeTests:176 Starting
> >> isolated execution on 54.157.22.221
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena <se...@cloudera.com>
> wrote:
> >>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
> >>> Available' status was set. I does not say at what time, but Jenkins
> runs a
> >>> job every 5 min to detect which files must be tested for all "Patch
> >>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue
> many
> >>> times.
> >>>
> >>> I would recommend to set the JIRA to 'Patch Available' when the patch
> is
> >>> reviewed to avoid many patches to be added to the queue.
> >>>
> >>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com>
> wrote:
> >>>
> >>>> Hi All
> >>>>
> >>>> I am noticing something weird in precommit queue. I can see
> >>>> HIVE-12541currenly running, but is also waiting next in the queue.
> The same
> >>>> jira ran few more times recently. I cannot see any other patches in
> the
> >>>> queue. Sometime I see bunch of patches in the queue but it will soon
> be
> >>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone
> seen
> >>>> this issue?
> >>>>
> >>>> Thanks
> >>>> Prasanth
>
Re: Precommit unstable?
Posted by Thejas Nair <th...@gmail.com>.
I killed the current 10+hr test run to see if next test would run
faster. (Discussed with Prasanth, the author of patch being tested).
However, I doubt if the next one is going to perform any better. If
we could try restarting the ptest2 system, I think that might be
better thing to do (instead of waiting several hours to realize that
the issue has not gone away).
On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair <th...@gmail.com> wrote:
> Would restart of jenkins or the hosts help with this ?
>
>
> On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair <th...@gmail.com> wrote:
>> The build running right now has been running for almost 10 hours.
>> I am wondering if there is something wrong with the hosts being used
>> right now. Some commands seem to be taking longer and I get some
>> intermittent errors from the jenkins UI.
>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
>> This rsync command seems to have taken a long time to finish -
>>
>> 2015-12-01 11:42:41,394 INFO LocalCommand.<init>:35 Starting timeout
>> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> --timeout 600 hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
>> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
>> 2015-12-01 12:54:52,522 INFO LocalCommand.<init>:35 Starting timeout
>> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> --timeout 600 hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
>> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
>> 2015-12-01 12:54:54,240 INFO HostExecutor.executeTests:176 Starting
>> isolated execution on 54.157.22.221
>>
>>
>>
>>
>>
>> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena <se...@cloudera.com> wrote:
>>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>>> Available' status was set. I does not say at what time, but Jenkins runs a
>>> job every 5 min to detect which files must be tested for all "Patch
>>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
>>> times.
>>>
>>> I would recommend to set the JIRA to 'Patch Available' when the patch is
>>> reviewed to avoid many patches to be added to the queue.
>>>
>>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com> wrote:
>>>
>>>> Hi All
>>>>
>>>> I am noticing something weird in precommit queue. I can see
>>>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>>>> jira ran few more times recently. I cannot see any other patches in the
>>>> queue. Sometime I see bunch of patches in the queue but it will soon be
>>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>>>> this issue?
>>>>
>>>> Thanks
>>>> Prasanth
Re: Precommit unstable?
Posted by Thejas Nair <th...@gmail.com>.
Would restart of jenkins or the hosts help with this ?
On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair <th...@gmail.com> wrote:
> The build running right now has been running for almost 10 hours.
> I am wondering if there is something wrong with the hosts being used
> right now. Some commands seem to be taking longer and I get some
> intermittent errors from the jenkins UI.
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
> This rsync command seems to have taken a long time to finish -
>
> 2015-12-01 11:42:41,394 INFO LocalCommand.<init>:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
> 2015-12-01 12:54:52,522 INFO LocalCommand.<init>:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
> 2015-12-01 12:54:54,240 INFO HostExecutor.executeTests:176 Starting
> isolated execution on 54.157.22.221
>
>
>
>
>
> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena <se...@cloudera.com> wrote:
>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> Available' status was set. I does not say at what time, but Jenkins runs a
>> job every 5 min to detect which files must be tested for all "Patch
>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
>> times.
>>
>> I would recommend to set the JIRA to 'Patch Available' when the patch is
>> reviewed to avoid many patches to be added to the queue.
>>
>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> I am noticing something weird in precommit queue. I can see
>>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>>> jira ran few more times recently. I cannot see any other patches in the
>>> queue. Sometime I see bunch of patches in the queue but it will soon be
>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>>> this issue?
>>>
>>> Thanks
>>> Prasanth
Re: Precommit unstable?
Posted by Thejas Nair <th...@gmail.com>.
The build running right now has been running for almost 10 hours.
I am wondering if there is something wrong with the hosts being used
right now. Some commands seem to be taking longer and I get some
intermittent errors from the jenkins UI.
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
This rsync command seems to have taken a long time to finish -
2015-12-01 11:42:41,394 INFO LocalCommand.<init>:35 Starting timeout
1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
--timeout 600 hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
2015-12-01 12:54:52,522 INFO LocalCommand.<init>:35 Starting timeout
1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
--timeout 600 hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
2015-12-01 12:54:54,240 INFO HostExecutor.executeTests:176 Starting
isolated execution on 54.157.22.221
On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena <se...@cloudera.com> wrote:
> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
> Available' status was set. I does not say at what time, but Jenkins runs a
> job every 5 min to detect which files must be tested for all "Patch
> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
> times.
>
> I would recommend to set the JIRA to 'Patch Available' when the patch is
> reviewed to avoid many patches to be added to the queue.
>
> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com> wrote:
>
>> Hi All
>>
>> I am noticing something weird in precommit queue. I can see
>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>> jira ran few more times recently. I cannot see any other patches in the
>> queue. Sometime I see bunch of patches in the queue but it will soon be
>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>> this issue?
>>
>> Thanks
>> Prasanth
Re: Precommit unstable?
Posted by Sergio Pena <se...@cloudera.com>.
I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
Available' status was set. I does not say at what time, but Jenkins runs a
job every 5 min to detect which files must be tested for all "Patch
Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
times.
I would recommend to set the JIRA to 'Patch Available' when the patch is
reviewed to avoid many patches to be added to the queue.
On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J <j....@gmail.com> wrote:
> Hi All
>
> I am noticing something weird in precommit queue. I can see
> HIVE-12541currenly running, but is also waiting next in the queue. The same
> jira ran few more times recently. I cannot see any other patches in the
> queue. Sometime I see bunch of patches in the queue but it will soon be
> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
> this issue?
>
> Thanks
> Prasanth