You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hari Sreekumar <hs...@clickable.com> on 2011/04/04 12:33:09 UTC

ZooKeeperConnectionException whenever table is flushed or majorcompacted

Hi,

I get this exception when I try to flush META using
HbaseAdmin.flush(".META."). I get the same exception when I do major
compact:

11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to hadoopqa2/192.168.1.50:2181, initiating session
11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
connection and attempting reconnect
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
        at sun.nio.ch.IOUtil.read(IOUtil.java:204)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server hadoopqa2/192.168.1.50:2181
11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to hadoopqa2/192.168.1.50:2181, initiating session
11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
connection and attempting reconnect
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
        at sun.nio.ch.IOUtil.read(IOUtil.java:204)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
11/04/04 21:26:32 FATAL
com.clickable.dataengine.hbase.create.CreateTableByXML:
*************************************************************
11/04/04 21:26:32 FATAL
com.clickable.dataengine.hbase.create.CreateTableByXML:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
11/04/04 21:26:32 FATAL
com.clickable.dataengine.hbase.create.CreateTableByXML:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)

11/04/04 21:26:32 FATAL
com.clickable.dataengine.hbase.create.CreateTableByXML:
*************************************************************
11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server hadoopqa2/192.168.1.50:2181
11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket connection
established to hadoopqa2/192.168.1.50:2181, initiating session
11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
additional data from server sessionid 0x0, likely server has closed socket,
closing socket connection and attempting reconnect

This doesn't happen always. I think it is happening sometimes only. I
suspect this has something to do with the asynchronous nature of these
commands. Any ideas?

Thanks,
Hari

Re: ZooKeeperConnectionException whenever table is flushed or majorcompacted

Posted by Jean-Daniel Cryans <jd...@apache.org>.
Nothing comes to mind as to why it would "fix" it, maybe I don't
understand what you did instead.

BTW I created https://issues.apache.org/jira/browse/HBASE-3734 to
track the issue.

J-D

On Mon, Apr 4, 2011 at 9:54 PM, Hari Sreekumar <hs...@clickable.com> wrote:
> Ah, I didn't notice it was happening for tableExists() in this instance. But
> it was always happening for flush() and majorCompact() methods earlier so I
> didn't check the log when copying it. So I thought it might have something
> to do with these methods. Yes, I see "Too many connections" error in the
> zookeeper logs just about the time I go the exception.
>
> Basically, I was using the same HBaseAdmin instance to create all the tables
> (There are about 15 tables). When I split it into groups of 5 tables with
> each HBaseAdmin instance, and then recreating an HBaseAdmin before going for
> the other tables, then I don't get this error. When I had done this, I was
> assuming something related to the asynchronous nature of flush(), but that
> doesn't make sense at all now. So why is it getting fixed (seems to) when I
> do this?
>
> Thanks,
> Hari
>
> On Mon, Apr 4, 2011 at 10:40 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> As far as I can tell the async nature of those operations has nothing
>> to do with what you see since it's not even able to get a session from
>> ZooKeeper (so it's not even talking to the region servers). If you
>> look at the stack trace:
>>
>>
>> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
>> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>>
>> It's not even happening on the call you said it did ;)
>>
>> But going through the code I see a major issue,
>> HBaseAdmin.getCatalogTracker creates new Configuration like if it was
>> free (it's not). Calling this method more than 30 times will probably
>> give you a "too many connections" in the zookeeper log (if you look at
>> the correct, what you pasted is 8 seconds later).
>>
>> I'll open a jira.
>>
>> J-D
>>
>> On Mon, Apr 4, 2011 at 3:43 AM, Hari Sreekumar <hs...@clickable.com>
>> wrote:
>> > I have this in the zookeeper logs, which might be helpful:
>> >
>> > 2011-04-04 21:26:40,356 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51467 which had sessionid 0x12f20e3713b02ee
>> > 2011-04-04 21:26:40,357 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> > EndOfStreamException: Unable to read additiona
>> > l data from client sessionid 0x12f20e3713b02d9, likely client has closed
>> > socket
>> > 2011-04-04 21:26:40,357 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51435 which had sessionid 0x12f20e3713b02d9
>> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> > EndOfStreamException: Unable to read additiona
>> > l data from client sessionid 0x12f20e3713b02e5, likely client has closed
>> > socket
>> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51453 which had sessionid 0x12f20e3713b02e5
>> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
>> > EndOfStreamException: Unable to read additiona
>> > l data from client sessionid 0x12f20e3713b02fd, likely client has closed
>> > socket
>> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
>> > Closed socket connection for client /192.168.1
>> > .49:51490 which had sessionid 0x12f20e3713b02fd
>> >
>> >
>> >
>> > On Mon, Apr 4, 2011 at 4:03 PM, Hari Sreekumar <hsreekumar@clickable.com
>> >wrote:
>> >
>> >> Hi,
>> >>
>> >> I get this exception when I try to flush META using
>> >> HbaseAdmin.flush(".META."). I get the same exception when I do major
>> >> compact:
>> >>
>> >> 11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection
>> >> established to hadoopqa2/192.168.1.50:2181, initiating session
>> >> 11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
>> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
>> >> connection and attempting reconnect
>> >> java.io.IOException: Connection reset by peer
>> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
>> >> connection to server hadoopqa2/192.168.1.50:2181
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection
>> >> established to hadoopqa2/192.168.1.50:2181, initiating session
>> >> 11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
>> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
>> >> connection and attempting reconnect
>> >> java.io.IOException: Connection reset by peer
>> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>> >>         at
>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >> *************************************************************
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >> KeeperErrorCode = ConnectionLoss for /hbase
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
>> >>
>> >>
>> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
>> >>
>> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>> >>
>> >>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>> >>
>> >>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
>> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> java.lang.reflect.Method.invoke(Method.java:597)
>> >> org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> >>
>> >> 11/04/04 21:26:32 FATAL
>> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> >> *************************************************************
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
>> >> connection to server hadoopqa2/192.168.1.50:2181
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
>> connection
>> >> established to hadoopqa2/192.168.1.50:2181, initiating session
>> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
>> >> additional data from server sessionid 0x0, likely server has closed
>> socket,
>> >> closing socket connection and attempting reconnect
>> >>
>> >> This doesn't happen always. I think it is happening sometimes only. I
>> >> suspect this has something to do with the asynchronous nature of these
>> >> commands. Any ideas?
>> >>
>> >> Thanks,
>> >> Hari
>> >>
>> >
>>
>

Re: ZooKeeperConnectionException whenever table is flushed or majorcompacted

Posted by Hari Sreekumar <hs...@clickable.com>.
Ah, I didn't notice it was happening for tableExists() in this instance. But
it was always happening for flush() and majorCompact() methods earlier so I
didn't check the log when copying it. So I thought it might have something
to do with these methods. Yes, I see "Too many connections" error in the
zookeeper logs just about the time I go the exception.

Basically, I was using the same HBaseAdmin instance to create all the tables
(There are about 15 tables). When I split it into groups of 5 tables with
each HBaseAdmin instance, and then recreating an HBaseAdmin before going for
the other tables, then I don't get this error. When I had done this, I was
assuming something related to the asynchronous nature of flush(), but that
doesn't make sense at all now. So why is it getting fixed (seems to) when I
do this?

Thanks,
Hari

On Mon, Apr 4, 2011 at 10:40 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:

> As far as I can tell the async nature of those operations has nothing
> to do with what you see since it's not even able to get a session from
> ZooKeeper (so it's not even talking to the region servers). If you
> look at the stack trace:
>
>
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>
> It's not even happening on the call you said it did ;)
>
> But going through the code I see a major issue,
> HBaseAdmin.getCatalogTracker creates new Configuration like if it was
> free (it's not). Calling this method more than 30 times will probably
> give you a "too many connections" in the zookeeper log (if you look at
> the correct, what you pasted is 8 seconds later).
>
> I'll open a jira.
>
> J-D
>
> On Mon, Apr 4, 2011 at 3:43 AM, Hari Sreekumar <hs...@clickable.com>
> wrote:
> > I have this in the zookeeper logs, which might be helpful:
> >
> > 2011-04-04 21:26:40,356 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51467 which had sessionid 0x12f20e3713b02ee
> > 2011-04-04 21:26:40,357 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > EndOfStreamException: Unable to read additiona
> > l data from client sessionid 0x12f20e3713b02d9, likely client has closed
> > socket
> > 2011-04-04 21:26:40,357 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51435 which had sessionid 0x12f20e3713b02d9
> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > EndOfStreamException: Unable to read additiona
> > l data from client sessionid 0x12f20e3713b02e5, likely client has closed
> > socket
> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51453 which had sessionid 0x12f20e3713b02e5
> > 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > EndOfStreamException: Unable to read additiona
> > l data from client sessionid 0x12f20e3713b02fd, likely client has closed
> > socket
> > 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /192.168.1
> > .49:51490 which had sessionid 0x12f20e3713b02fd
> >
> >
> >
> > On Mon, Apr 4, 2011 at 4:03 PM, Hari Sreekumar <hsreekumar@clickable.com
> >wrote:
> >
> >> Hi,
> >>
> >> I get this exception when I try to flush META using
> >> HbaseAdmin.flush(".META."). I get the same exception when I do major
> >> compact:
> >>
> >> 11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection
> >> established to hadoopqa2/192.168.1.50:2181, initiating session
> >> 11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
> >> connection and attempting reconnect
> >> java.io.IOException: Connection reset by peer
> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> >> connection to server hadoopqa2/192.168.1.50:2181
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection
> >> established to hadoopqa2/192.168.1.50:2181, initiating session
> >> 11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> >> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
> >> connection and attempting reconnect
> >> java.io.IOException: Connection reset by peer
> >>         at sun.nio.ch.FileDispatcher.read0(Native Method)
> >>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> >>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
> >>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
> >>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
> >>         at
> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >> *************************************************************
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> >> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >> KeeperErrorCode = ConnectionLoss for /hbase
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
> >>
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
> >>
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
> >>
> >>
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
> >>
> >>
> com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> java.lang.reflect.Method.invoke(Method.java:597)
> >> org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>
> >> 11/04/04 21:26:32 FATAL
> >> com.clickable.dataengine.hbase.create.CreateTableByXML:
> >> *************************************************************
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> >> connection to server hadoopqa2/192.168.1.50:2181
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection
> >> established to hadoopqa2/192.168.1.50:2181, initiating session
> >> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
> >> additional data from server sessionid 0x0, likely server has closed
> socket,
> >> closing socket connection and attempting reconnect
> >>
> >> This doesn't happen always. I think it is happening sometimes only. I
> >> suspect this has something to do with the asynchronous nature of these
> >> commands. Any ideas?
> >>
> >> Thanks,
> >> Hari
> >>
> >
>

Re: ZooKeeperConnectionException whenever table is flushed or majorcompacted

Posted by Jean-Daniel Cryans <jd...@apache.org>.
As far as I can tell the async nature of those operations has nothing
to do with what you see since it's not even able to get a session from
ZooKeeper (so it's not even talking to the region servers). If you
look at the stack trace:

org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)

It's not even happening on the call you said it did ;)

But going through the code I see a major issue,
HBaseAdmin.getCatalogTracker creates new Configuration like if it was
free (it's not). Calling this method more than 30 times will probably
give you a "too many connections" in the zookeeper log (if you look at
the correct, what you pasted is 8 seconds later).

I'll open a jira.

J-D

On Mon, Apr 4, 2011 at 3:43 AM, Hari Sreekumar <hs...@clickable.com> wrote:
> I have this in the zookeeper logs, which might be helpful:
>
> 2011-04-04 21:26:40,356 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /192.168.1
> .49:51467 which had sessionid 0x12f20e3713b02ee
> 2011-04-04 21:26:40,357 WARN org.apache.zookeeper.server.NIOServerCnxn:
> EndOfStreamException: Unable to read additiona
> l data from client sessionid 0x12f20e3713b02d9, likely client has closed
> socket
> 2011-04-04 21:26:40,357 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /192.168.1
> .49:51435 which had sessionid 0x12f20e3713b02d9
> 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
> EndOfStreamException: Unable to read additiona
> l data from client sessionid 0x12f20e3713b02e5, likely client has closed
> socket
> 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /192.168.1
> .49:51453 which had sessionid 0x12f20e3713b02e5
> 2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
> EndOfStreamException: Unable to read additiona
> l data from client sessionid 0x12f20e3713b02fd, likely client has closed
> socket
> 2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /192.168.1
> .49:51490 which had sessionid 0x12f20e3713b02fd
>
>
>
> On Mon, Apr 4, 2011 at 4:03 PM, Hari Sreekumar <hs...@clickable.com>wrote:
>
>> Hi,
>>
>> I get this exception when I try to flush META using
>> HbaseAdmin.flush(".META."). I get the same exception when I do major
>> compact:
>>
>> 11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket connection
>> established to hadoopqa2/192.168.1.50:2181, initiating session
>> 11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
>> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
>> connection and attempting reconnect
>> java.io.IOException: Connection reset by peer
>>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>         at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>>         at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
>> connection to server hadoopqa2/192.168.1.50:2181
>> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket connection
>> established to hadoopqa2/192.168.1.50:2181, initiating session
>> 11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
>> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
>> connection and attempting reconnect
>> java.io.IOException: Connection reset by peer
>>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>         at
>> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>>         at
>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
>> 11/04/04 21:26:32 FATAL
>> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> *************************************************************
>> 11/04/04 21:26:32 FATAL
>> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> org.apache.hadoop.hbase.ZooKeeperConnectionException:
>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> KeeperErrorCode = ConnectionLoss for /hbase
>> 11/04/04 21:26:32 FATAL
>> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
>>
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
>>
>> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
>>
>> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
>> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>>
>> com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> java.lang.reflect.Method.invoke(Method.java:597)
>> org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>> 11/04/04 21:26:32 FATAL
>> com.clickable.dataengine.hbase.create.CreateTableByXML:
>> *************************************************************
>> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
>> connection to server hadoopqa2/192.168.1.50:2181
>> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket connection
>> established to hadoopqa2/192.168.1.50:2181, initiating session
>> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
>> additional data from server sessionid 0x0, likely server has closed socket,
>> closing socket connection and attempting reconnect
>>
>> This doesn't happen always. I think it is happening sometimes only. I
>> suspect this has something to do with the asynchronous nature of these
>> commands. Any ideas?
>>
>> Thanks,
>> Hari
>>
>

Re: ZooKeeperConnectionException whenever table is flushed or majorcompacted

Posted by Hari Sreekumar <hs...@clickable.com>.
I have this in the zookeeper logs, which might be helpful:

2011-04-04 21:26:40,356 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /192.168.1
.49:51467 which had sessionid 0x12f20e3713b02ee
2011-04-04 21:26:40,357 WARN org.apache.zookeeper.server.NIOServerCnxn:
EndOfStreamException: Unable to read additiona
l data from client sessionid 0x12f20e3713b02d9, likely client has closed
socket
2011-04-04 21:26:40,357 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /192.168.1
.49:51435 which had sessionid 0x12f20e3713b02d9
2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
EndOfStreamException: Unable to read additiona
l data from client sessionid 0x12f20e3713b02e5, likely client has closed
socket
2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /192.168.1
.49:51453 which had sessionid 0x12f20e3713b02e5
2011-04-04 21:26:40,358 WARN org.apache.zookeeper.server.NIOServerCnxn:
EndOfStreamException: Unable to read additiona
l data from client sessionid 0x12f20e3713b02fd, likely client has closed
socket
2011-04-04 21:26:40,358 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /192.168.1
.49:51490 which had sessionid 0x12f20e3713b02fd



On Mon, Apr 4, 2011 at 4:03 PM, Hari Sreekumar <hs...@clickable.com>wrote:

> Hi,
>
> I get this exception when I try to flush META using
> HbaseAdmin.flush(".META."). I get the same exception when I do major
> compact:
>
> 11/04/04 21:26:31 INFO org.apache.zookeeper.ClientCnxn: Socket connection
> established to hadoopqa2/192.168.1.50:2181, initiating session
> 11/04/04 21:26:31 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
> connection and attempting reconnect
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>         at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>         at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server hadoopqa2/192.168.1.50:2181
> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket connection
> established to hadoopqa2/192.168.1.50:2181, initiating session
> 11/04/04 21:26:32 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> server hadoopqa2/192.168.1.50:2181, unexpected error, closing socket
> connection and attempting reconnect
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcher.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:204)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>         at
> org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
>         at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
> 11/04/04 21:26:32 FATAL
> com.clickable.dataengine.hbase.create.CreateTableByXML:
> *************************************************************
> 11/04/04 21:26:32 FATAL
> com.clickable.dataengine.hbase.create.CreateTableByXML:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> 11/04/04 21:26:32 FATAL
> com.clickable.dataengine.hbase.create.CreateTableByXML:
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:292)
>
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
>
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:104)
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:159)
>
> com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(CreateTableByXML.java:144)
>
> com.clickable.dataengine.hbase.create.CreateTableByXML.main(CreateTableByXML.java:68)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> 11/04/04 21:26:32 FATAL
> com.clickable.dataengine.hbase.create.CreateTableByXML:
> *************************************************************
> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server hadoopqa2/192.168.1.50:2181
> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Socket connection
> established to hadoopqa2/192.168.1.50:2181, initiating session
> 11/04/04 21:26:32 INFO org.apache.zookeeper.ClientCnxn: Unable to read
> additional data from server sessionid 0x0, likely server has closed socket,
> closing socket connection and attempting reconnect
>
> This doesn't happen always. I think it is happening sometimes only. I
> suspect this has something to do with the asynchronous nature of these
> commands. Any ideas?
>
> Thanks,
> Hari
>