You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2016/04/06 22:29:52 UTC
recent metastore failures in HiveQA
Has anyone else noticed that many tests that involve metastore started
failing lately? The failures are sporadic and happen both in the tests
that test metastore, and q files that use metastore…
The error is always something like
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
ava:198)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
Utils.java:1208)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
ls.java:1195)
at
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
etrics.java:54)
I wonder if someone has insight on whether this is an environment issue,
or someone broke something recently, before we investigate more :)
Re: recent metastore failures in HiveQA
Posted by Wei Zheng <wz...@hortonworks.com>.
Still seeing lots of failing tests with "Connection refused" message.. But locally they run fine.
Thanks,
Wei
On 4/12/16, 14:48, "Ashutosh Chauhan" <ha...@apache.org> wrote:
>I can repro locally hang of TestJdbcWithMiniHS2 and I also saw it hanging
>on recent QA runs:
>http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7556/failed/TestJdbcWithMiniHS2/
>
>Wondering if you guys have seen this?
>
>Thanks,
>Ashutosh
>
>On Tue, Apr 12, 2016 at 11:28 AM, Szehon Ho <sz...@cloudera.com> wrote:
>
>> Thanks Thejas for this patch!
>>
>> I'm also going to restart PTest and force recreation of the test slaves
>> based on fresh image to see if it resolves the issue (in case the test
>> slaves are getting too loaded and slow to start HMS in time). If not then
>> Thejas's patch should tell us a bit more.
>>
>> On Tue, Apr 12, 2016 at 12:35 AM, Thejas Nair <th...@gmail.com>
>> wrote:
>>
>> > Created a patch that should hopefully help in figuring out whats going
>> > on - https://issues.apache.org/jira/browse/HIVE-13491
>> >
>> >
>> > On Wed, Apr 6, 2016 at 1:56 PM, Szehon Ho <sz...@cloudera.com> wrote:
>> > > Yea thanks for point it out. I see it too and am not able to reproduce
>> > it
>> > > locally. It points to an environment issue, but not aware anything
>> > changed
>> > > with the environment.
>> > >
>> > > Anyone have any ideas?
>> > >
>> > > On Wed, Apr 6, 2016 at 1:29 PM, Sergey Shelukhin <
>> sergey@hortonworks.com
>> > >
>> > > wrote:
>> > >
>> > >> Has anyone else noticed that many tests that involve metastore started
>> > >> failing lately? The failures are sporadic and happen both in the tests
>> > >> that test metastore, and q files that use metastore…
>> > >> The error is always something like
>> > >> java.net.ConnectException: Connection refused
>> > >> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> > >> at
>> > >>
>> >
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
>> > >> )
>> > >> at
>> > >>
>> >
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
>> > >> ava:198)
>> > >> at
>> > >>
>> >
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>> > >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> > >> at java.net.Socket.connect(Socket.java:579)
>> > >> at
>> > >>
>> >
>> org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
>> > >> Utils.java:1208)
>> > >> at
>> > >>
>> >
>> org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
>> > >> ls.java:1195)
>> > >> at
>> > >>
>> >
>> org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
>> > >> etrics.java:54)
>> > >>
>> > >> I wonder if someone has insight on whether this is an environment
>> issue,
>> > >> or someone broke something recently, before we investigate more :)
>> > >>
>> > >>
>> > >>
>> >
>>
Re: recent metastore failures in HiveQA
Posted by Ashutosh Chauhan <ha...@apache.org>.
I can repro locally hang of TestJdbcWithMiniHS2 and I also saw it hanging
on recent QA runs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7556/failed/TestJdbcWithMiniHS2/
Wondering if you guys have seen this?
Thanks,
Ashutosh
On Tue, Apr 12, 2016 at 11:28 AM, Szehon Ho <sz...@cloudera.com> wrote:
> Thanks Thejas for this patch!
>
> I'm also going to restart PTest and force recreation of the test slaves
> based on fresh image to see if it resolves the issue (in case the test
> slaves are getting too loaded and slow to start HMS in time). If not then
> Thejas's patch should tell us a bit more.
>
> On Tue, Apr 12, 2016 at 12:35 AM, Thejas Nair <th...@gmail.com>
> wrote:
>
> > Created a patch that should hopefully help in figuring out whats going
> > on - https://issues.apache.org/jira/browse/HIVE-13491
> >
> >
> > On Wed, Apr 6, 2016 at 1:56 PM, Szehon Ho <sz...@cloudera.com> wrote:
> > > Yea thanks for point it out. I see it too and am not able to reproduce
> > it
> > > locally. It points to an environment issue, but not aware anything
> > changed
> > > with the environment.
> > >
> > > Anyone have any ideas?
> > >
> > > On Wed, Apr 6, 2016 at 1:29 PM, Sergey Shelukhin <
> sergey@hortonworks.com
> > >
> > > wrote:
> > >
> > >> Has anyone else noticed that many tests that involve metastore started
> > >> failing lately? The failures are sporadic and happen both in the tests
> > >> that test metastore, and q files that use metastore…
> > >> The error is always something like
> > >> java.net.ConnectException: Connection refused
> > >> at java.net.PlainSocketImpl.socketConnect(Native Method)
> > >> at
> > >>
> >
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
> > >> )
> > >> at
> > >>
> >
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
> > >> ava:198)
> > >> at
> > >>
> >
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> > >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > >> at java.net.Socket.connect(Socket.java:579)
> > >> at
> > >>
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
> > >> Utils.java:1208)
> > >> at
> > >>
> >
> org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
> > >> ls.java:1195)
> > >> at
> > >>
> >
> org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
> > >> etrics.java:54)
> > >>
> > >> I wonder if someone has insight on whether this is an environment
> issue,
> > >> or someone broke something recently, before we investigate more :)
> > >>
> > >>
> > >>
> >
>
Re: recent metastore failures in HiveQA
Posted by Szehon Ho <sz...@cloudera.com>.
Thanks Thejas for this patch!
I'm also going to restart PTest and force recreation of the test slaves
based on fresh image to see if it resolves the issue (in case the test
slaves are getting too loaded and slow to start HMS in time). If not then
Thejas's patch should tell us a bit more.
On Tue, Apr 12, 2016 at 12:35 AM, Thejas Nair <th...@gmail.com> wrote:
> Created a patch that should hopefully help in figuring out whats going
> on - https://issues.apache.org/jira/browse/HIVE-13491
>
>
> On Wed, Apr 6, 2016 at 1:56 PM, Szehon Ho <sz...@cloudera.com> wrote:
> > Yea thanks for point it out. I see it too and am not able to reproduce
> it
> > locally. It points to an environment issue, but not aware anything
> changed
> > with the environment.
> >
> > Anyone have any ideas?
> >
> > On Wed, Apr 6, 2016 at 1:29 PM, Sergey Shelukhin <sergey@hortonworks.com
> >
> > wrote:
> >
> >> Has anyone else noticed that many tests that involve metastore started
> >> failing lately? The failures are sporadic and happen both in the tests
> >> that test metastore, and q files that use metastore…
> >> The error is always something like
> >> java.net.ConnectException: Connection refused
> >> at java.net.PlainSocketImpl.socketConnect(Native Method)
> >> at
> >>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
> >> )
> >> at
> >>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
> >> ava:198)
> >> at
> >>
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> >> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >> at java.net.Socket.connect(Socket.java:579)
> >> at
> >>
> org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
> >> Utils.java:1208)
> >> at
> >>
> org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
> >> ls.java:1195)
> >> at
> >>
> org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
> >> etrics.java:54)
> >>
> >> I wonder if someone has insight on whether this is an environment issue,
> >> or someone broke something recently, before we investigate more :)
> >>
> >>
> >>
>
Re: recent metastore failures in HiveQA
Posted by Thejas Nair <th...@gmail.com>.
Created a patch that should hopefully help in figuring out whats going
on - https://issues.apache.org/jira/browse/HIVE-13491
On Wed, Apr 6, 2016 at 1:56 PM, Szehon Ho <sz...@cloudera.com> wrote:
> Yea thanks for point it out. I see it too and am not able to reproduce it
> locally. It points to an environment issue, but not aware anything changed
> with the environment.
>
> Anyone have any ideas?
>
> On Wed, Apr 6, 2016 at 1:29 PM, Sergey Shelukhin <se...@hortonworks.com>
> wrote:
>
>> Has anyone else noticed that many tests that involve metastore started
>> failing lately? The failures are sporadic and happen both in the tests
>> that test metastore, and q files that use metastore…
>> The error is always something like
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
>> )
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
>> ava:198)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:579)
>> at
>> org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
>> Utils.java:1208)
>> at
>> org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
>> ls.java:1195)
>> at
>> org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
>> etrics.java:54)
>>
>> I wonder if someone has insight on whether this is an environment issue,
>> or someone broke something recently, before we investigate more :)
>>
>>
>>
Re: recent metastore failures in HiveQA
Posted by Szehon Ho <sz...@cloudera.com>.
Yea thanks for point it out. I see it too and am not able to reproduce it
locally. It points to an environment issue, but not aware anything changed
with the environment.
Anyone have any ideas?
On Wed, Apr 6, 2016 at 1:29 PM, Sergey Shelukhin <se...@hortonworks.com>
wrote:
> Has anyone else noticed that many tests that involve metastore started
> failing lately? The failures are sporadic and happen both in the tests
> that test metastore, and q files that use metastore…
> The error is always something like
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339
> )
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.j
> ava:198)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.loopUntilHMSReady(MetaStore
> Utils.java:1208)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.startMetaStore(MetaStoreUti
> ls.java:1195)
> at
> org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.before(TestMetaStoreM
> etrics.java:54)
>
> I wonder if someone has insight on whether this is an environment issue,
> or someone broke something recently, before we investigate more :)
>
>
>