You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Rajeshkumar J <ra...@gmail.com> on 2016/01/10 17:56:14 UTC
Hbase thrift server issues
Hi,
I am connecting via Hbase thrift server to access records in Hbase and I am
doing this in C# and i am using range scan to get records from the table.
This seems to be working for few hours but after few hours it didn't
returned any records and then i searched hbase logs i found this
2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x3520b1721650006
2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper:
Session: 0x3520b1721650006 closed
2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
zookeeper.ClientCnxn: EventThread shut down
And I am using hbase 0.98.13
I have asked this question before and some user suggested me to upgrade to
new version so i have upgraded HBase to 1.1.2. But now it also it didn't
return any records but different log generated as shown below
2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(xxxx:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout =
40000
2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
client.ConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x1522613893e0005
2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
zookeeper.ClientCnxn: EventThread shut down
Can any one help me in this?
Regards,
Rajeshkumar J
Re: Hbase thrift server issues
Posted by Rajeshkumar J <ra...@gmail.com>.
Hi Ted,
I have taken stacktrace of Hbase thrift server. Plz find it in below
location
http://pastebin.com/QQjZhi36
Thanks
On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yu...@gmail.com> wrote:
> When you search for 'java jstack tutorial' using google, you would find a
> lot of tutorials.
>
> You can use 'ps aux' command to search for the following class:
> org.apache.hadoop.hbase.thrift.ThriftServer
>
> Apply jstack on the corresponding process id.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > I don't know how how to take stacktrace using jstack. But i tried
> using
> > some commands but it fails. can you help me in this?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > What I meant was to capture stack trace of the thrift server (using
> > jstack
> > > command) when your query is running.
> > >
> > > This would give us some more clue as to the state thrift server was in
> > when
> > > the problem happened.
> > >
> > > Cheers
> > >
> > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > This is what I found in the Hbase thrift server logs
> > > >
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.library.path=;C:\Hadoop\bin
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.compiler=<NA>
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.name=Windows Server 2012
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.arch=amd64
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.version=6.2
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.name=xxxx
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.home=C:\
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.dir=C:\HBase\bin
> > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 06:27:43,339 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 06:27:43,342 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 06:27:43,365 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 07:14:10,607 INFO
> > > > [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > > Monitor missed its start time
> > > > 2016-01-09 07:16:43,511 INFO
> > > > [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > DATA_BLOCK_ENCODING
> > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS
> =>
> > > '1',
> > > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > > 'false',
> > > > BLOCKCACHE => 'true'}
> > > > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> CreateTableProcedure
> > > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > > > exec-time=462msec
> > exception=org.apache.hadoop.hbase.TableExistsException:
> > > > chk2
> > > > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0004
> > > > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > > > zookeeper.RecoverableZooKeeper: Process
> > identifier=hconnection-0x591287f8
> > > > connecting to ZooKeeper
> > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > 2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 18:40:29,553 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 18:40:29,559 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 18:40:29,569 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x35226137e300008
> > > > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > > 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> > > > zookeeper.RecoverableZooKeeper: Process
> > identifier=hconnection-0x6305dbeb
> > > > connecting to ZooKeeper
> > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > 2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 20:05:42,507 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 20:05:42,510 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 20:05:42,708 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0005
> > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > >
> > > >
> > > > Thanks,
> > > > Rajeshkumar J
> > > >
> > > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > Can you take a stack trace of the thrift server and pastebin the
> > trace
> > > ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > > rajeshkumarit8292@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > I am connecting via Hbase thrift server to access records in
> Hbase
> > > and
> > > > I
> > > > > am
> > > > > > doing this in C# and i am using range scan to get records from
> the
> > > > table.
> > > > > > This seems to be working for few hours but after few hours it
> > didn't
> > > > > > returned any records and then i searched hbase logs i found this
> > > > > >
> > > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > > zookeeper
> > > > > > sessionid=0x3520b1721650006
> > > > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
> > > zookeeper.ZooKeeper:
> > > > > > Session: 0x3520b1721650006 closed
> > > > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > >
> > > > > > And I am using hbase 0.98.13
> > > > > >
> > > > > > I have asked this question before and some user suggested me to
> > > upgrade
> > > > > to
> > > > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > > > didn't
> > > > > > return any records but different log generated as shown below
> > > > > >
> > > > > > 2016-01-09 20:05:42,708 INFO
> > > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > > timeout =
> > > > > > 40000
> > > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > > sessionid=0x1522613893e0005
> > > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > >
> > > > > > Can any one help me in this?
> > > > > >
> > > > > > Regards,
> > > > > > Rajeshkumar J
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Hbase thrift server issues
Posted by Ted Yu <yu...@gmail.com>.
You can get the source code from 0.98 branch of:
https://github.com/apache/hbase
0.98.17 hasn't been released yet.
RC would come out this month.
If you cannot wait, you can build locally and deploy.
Cheers
> On Jan 11, 2016, at 11:30 PM, Rajeshkumar J <ra...@gmail.com> wrote:
>
> Hi Ted,
>
> As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have
> googled it for both its source code and binary but I didn't find one. can
> you guide me where I can find Hbase 0.98.17 source code or binary package
>
> Thanks
>
>> On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> I took a look at the stack trace you posted.
>>
>> Some thrift worker was in the state shown in HBASE-14926
>>
>> I wonder if the fix for HBASE-14926 would help in your case.
>>
>> Cheers
>>
>> On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J <
>> rajeshkumarit8292@gmail.com>
>> wrote:
>>
>>> Hi Ted,
>>>
>>> Plz ignore my previous mail as the log is taken when hbase is working
>>> fine. Please find the logs below when Hbase didn't return any records
>>>
>>> http://pastebin.com/APYjiGSP
>>>
>>> Thanks
>>>
>>>> On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>> When you search for 'java jstack tutorial' using google, you would
>> find a
>>>> lot of tutorials.
>>>>
>>>> You can use 'ps aux' command to search for the following class:
>>>> org.apache.hadoop.hbase.thrift.ThriftServer
>>>>
>>>> Apply jstack on the corresponding process id.
>>>>
>>>> Cheers
>>>>
>>>> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
>>>> rajeshkumarit8292@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Ted,
>>>>>
>>>>> I don't know how how to take stacktrace using jstack. But i tried
>>>> using
>>>>> some commands but it fails. can you help me in this?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>>>>>
>>>>>> What I meant was to capture stack trace of the thrift server (using
>>>>> jstack
>>>>>> command) when your query is running.
>>>>>>
>>>>>> This would give us some more clue as to the state thrift server was
>>> in
>>>>> when
>>>>>> the problem happened.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
>>>>>> rajeshkumarit8292@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Ted,
>>>>>>>
>>>>>>> This is what I found in the Hbase thrift server logs
>>>>>>>
>>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:java.library.path=;C:\Hadoop\bin
>>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:java.io.tmpdir=C:\Windows\TEMP\
>>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:java.compiler=<NA>
>>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:os.name=Windows Server 2012
>>>>>>> 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:os.arch=amd64
>>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:os.version=6.2
>>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:user.name=xxxx
>>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:user.home=C:\
>>>>>>> 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>> Client
>>>>>>> environment:user.dir=C:\HBase\bin
>>>>>>> 2016-01-09 06:27:43,142 INFO [thrift-worker-0]
>>> zookeeper.ZooKeeper:
>>>>>>> Initiating client connection,
>>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181
>>>>>>> sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
>>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181,
>> baseZNode=/hbase
>>>>>>> 2016-01-09 06:27:43,339 INFO
>>>>> [thrift-worker-0-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Opening socket connection to server
>>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
>>> SASL
>>>>>>> (unknown error)
>>>>>>> 2016-01-09 06:27:43,342 INFO
>>>>> [thrift-worker-0-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Socket connection established to
>>>>>>> hostname/xx.xx.x.xx:2181, initiating session
>>>>>>> 2016-01-09 06:27:43,365 INFO
>>>>> [thrift-worker-0-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Session establishment complete on server
>>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004,
>> negotiated
>>>>>> timeout
>>>>>>> = 40000
>>>>>>> 2016-01-09 07:14:10,607 INFO
>>>>>>> [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
>>>>>>> master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager
>>> Timeout
>>>>>>> Monitor missed its start time
>>>>>>> 2016-01-09 07:16:43,511 INFO
>>>>>>> [B.defaultRpcServer.handler=8,queue=2,port=60000]
>> master.HMaster:
>>>>>>> Client=Administrator/null create 'chk2', {NAME => 'fa',
>>>>>> DATA_BLOCK_ENCODING
>>>>>>> => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
>> VERSIONS
>>>> =>
>>>>>> '1',
>>>>>>> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
>>>>>>> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
>>>>>> 'false',
>>>>>>> BLOCKCACHE => 'true'}
>>>>>>> 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
>>>>>>> procedure2.ProcedureExecutor: Rolledback procedure
>>>> CreateTableProcedure
>>>>>>> (table=chk2) user=Administrator (auth:SIMPLE) id=2
>> state=ROLLEDBACK
>>>>>>> exec-time=462msec
>>>>> exception=org.apache.hadoop.hbase.TableExistsException:
>>>>>>> chk2
>>>>>>> 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>> zookeeper
>>>>>>> sessionid=0x1522613893e0004
>>>>>>> 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
>>>>>>> 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>> 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
>>>>>>> zookeeper.RecoverableZooKeeper: Process
>>>>> identifier=hconnection-0x591287f8
>>>>>>> connecting to ZooKeeper
>>>>>> ensemble=hostname:2181,hostname:2181,hostname:2181
>>>>>>> 2016-01-09 18:40:29,482 INFO [thrift-worker-1]
>>> zookeeper.ZooKeeper:
>>>>>>> Initiating client connection,
>>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181
>>>>>>> sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
>>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181,
>> baseZNode=/hbase
>>>>>>> 2016-01-09 18:40:29,553 INFO
>>>>> [thrift-worker-1-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Opening socket connection to server
>>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
>>> SASL
>>>>>>> (unknown error)
>>>>>>> 2016-01-09 18:40:29,559 INFO
>>>>> [thrift-worker-1-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Socket connection established to
>>>>>>> hostname/xx.xx.x.xx:2181, initiating session
>>>>>>> 2016-01-09 18:40:29,569 INFO
>>>>> [thrift-worker-1-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Session establishment complete on server
>>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008,
>> negotiated
>>>>>> timeout
>>>>>>> = 40000
>>>>>>> 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>> zookeeper
>>>>>>> sessionid=0x35226137e300008
>>>>>>> 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
>>>>>>> zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
>>>>>>> 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>> 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
>>>>>>> zookeeper.RecoverableZooKeeper: Process
>>>>> identifier=hconnection-0x6305dbeb
>>>>>>> connecting to ZooKeeper
>>>>>> ensemble=hostname:2181,hostname:2181,hostname:2181
>>>>>>> 2016-01-09 20:05:42,476 INFO [thrift-worker-13]
>>> zookeeper.ZooKeeper:
>>>>>>> Initiating client connection,
>>>>>>> connectString=hostname:2181,hostname:2181,hostname:2181
>>>>>>> sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
>>>>>>> quorum=hostname:2181,hostname:2181,hostname:2181,
>> baseZNode=/hbase
>>>>>>> 2016-01-09 20:05:42,507 INFO
>>>>>> [thrift-worker-13-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Opening socket connection to server
>>>>>>> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
>>> SASL
>>>>>>> (unknown error)
>>>>>>> 2016-01-09 20:05:42,510 INFO
>>>>>> [thrift-worker-13-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Socket connection established to
>>>>>>> hostname/xx.xx.x.xx:2181, initiating session
>>>>>>> 2016-01-09 20:05:42,708 INFO
>>>>>> [thrift-worker-13-SendThread(hostname:2181)]
>>>>>>> zookeeper.ClientCnxn: Session establishment complete on server
>>>>>>> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005,
>> negotiated
>>>>>> timeout
>>>>>>> = 40000
>>>>>>> 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>> zookeeper
>>>>>>> sessionid=0x1522613893e0005
>>>>>>> 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rajeshkumar J
>>>>>>>
>>>>>>> On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com>
>>>> wrote:
>>>>>>>
>>>>>>>> Can you take a stack trace of the thrift server and pastebin
>> the
>>>>> trace
>>>>>> ?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
>>>>>>>> rajeshkumarit8292@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am connecting via Hbase thrift server to access records in
>>>> Hbase
>>>>>> and
>>>>>>> I
>>>>>>>> am
>>>>>>>>> doing this in C# and i am using range scan to get records
>> from
>>>> the
>>>>>>> table.
>>>>>>>>> This seems to be working for few hours but after few hours it
>>>>> didn't
>>>>>>>>> returned any records and then i searched hbase logs i found
>>> this
>>>>>>>>>
>>>>>>>>> 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
>>>>>>>>> client.HConnectionManager$HConnectionImplementation: Closing
>>>>>> zookeeper
>>>>>>>>> sessionid=0x3520b1721650006
>>>>>>>>> 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
>>>>>> zookeeper.ZooKeeper:
>>>>>>>>> Session: 0x3520b1721650006 closed
>>>>>>>>> 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
>>>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>>>>
>>>>>>>>> And I am using hbase 0.98.13
>>>>>>>>>
>>>>>>>>> I have asked this question before and some user suggested me
>> to
>>>>>> upgrade
>>>>>>>> to
>>>>>>>>> new version so i have upgraded HBase to 1.1.2. But now it
>> also
>>> it
>>>>>>> didn't
>>>>>>>>> return any records but different log generated as shown below
>>>>>>>>>
>>>>>>>>> 2016-01-09 20:05:42,708 INFO
>>>>>> [thrift-worker-13-SendThread(xxxx:2181)]
>>>>>>>>> zookeeper.ClientCnxn: Session establishment complete on
>> server
>>>>>>>>> xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005,
>> negotiated
>>>>>>> timeout =
>>>>>>>>> 40000
>>>>>>>>> 2016-01-09 20:15:45,541 INFO
>> [ConnectionCache_ChoreService_1]
>>>>>>>>> client.ConnectionManager$HConnectionImplementation: Closing
>>>>> zookeeper
>>>>>>>>> sessionid=0x1522613893e0005
>>>>>>>>> 2016-01-09 20:15:45,567 INFO
>> [ConnectionCache_ChoreService_1]
>>>>>>>>> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
>>>>>>>>> 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
>>>>>>>>> zookeeper.ClientCnxn: EventThread shut down
>>>>>>>>>
>>>>>>>>> Can any one help me in this?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Rajeshkumar J
>>
Re: Hbase thrift server issues
Posted by Rajeshkumar J <ra...@gmail.com>.
Hi Ted,
As mentioned in that HBASE-14926 it is fixed in HBase 0.98.17. I have
googled it for both its source code and binary but I didn't find one. can
you guide me where I can find Hbase 0.98.17 source code or binary package
Thanks
On Tue, Jan 12, 2016 at 12:57 AM, Ted Yu <yu...@gmail.com> wrote:
> I took a look at the stack trace you posted.
>
> Some thrift worker was in the state shown in HBASE-14926
>
> I wonder if the fix for HBASE-14926 would help in your case.
>
> Cheers
>
> On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > Plz ignore my previous mail as the log is taken when hbase is working
> > fine. Please find the logs below when Hbase didn't return any records
> >
> > http://pastebin.com/APYjiGSP
> >
> > Thanks
> >
> > On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > When you search for 'java jstack tutorial' using google, you would
> find a
> > > lot of tutorials.
> > >
> > > You can use 'ps aux' command to search for the following class:
> > > org.apache.hadoop.hbase.thrift.ThriftServer
> > >
> > > Apply jstack on the corresponding process id.
> > >
> > > Cheers
> > >
> > > On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > I don't know how how to take stacktrace using jstack. But i tried
> > > using
> > > > some commands but it fails. can you help me in this?
> > > >
> > > > Thanks
> > > >
> > > > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > What I meant was to capture stack trace of the thrift server (using
> > > > jstack
> > > > > command) when your query is running.
> > > > >
> > > > > This would give us some more clue as to the state thrift server was
> > in
> > > > when
> > > > > the problem happened.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > > > rajeshkumarit8292@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Ted,
> > > > > >
> > > > > > This is what I found in the Hbase thrift server logs
> > > > > >
> > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:java.library.path=;C:\Hadoop\bin
> > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:java.compiler=<NA>
> > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:os.name=Windows Server 2012
> > > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:os.arch=amd64
> > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:os.version=6.2
> > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:user.name=xxxx
> > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:user.home=C:\
> > > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > Client
> > > > > > environment:user.dir=C:\HBase\bin
> > > > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0]
> > zookeeper.ZooKeeper:
> > > > > > Initiating client connection,
> > > > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > > > > quorum=hostname:2181,hostname:2181,hostname:2181,
> baseZNode=/hbase
> > > > > > 2016-01-09 06:27:43,339 INFO
> > > > [thrift-worker-0-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
> > SASL
> > > > > > (unknown error)
> > > > > > 2016-01-09 06:27:43,342 INFO
> > > > [thrift-worker-0-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Socket connection established to
> > > > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > > > 2016-01-09 06:27:43,365 INFO
> > > > [thrift-worker-0-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004,
> negotiated
> > > > > timeout
> > > > > > = 40000
> > > > > > 2016-01-09 07:14:10,607 INFO
> > > > > > [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager
> > Timeout
> > > > > > Monitor missed its start time
> > > > > > 2016-01-09 07:16:43,511 INFO
> > > > > > [B.defaultRpcServer.handler=8,queue=2,port=60000]
> master.HMaster:
> > > > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > > > DATA_BLOCK_ENCODING
> > > > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
> VERSIONS
> > > =>
> > > > > '1',
> > > > > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > > > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > > > > 'false',
> > > > > > BLOCKCACHE => 'true'}
> > > > > > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > > CreateTableProcedure
> > > > > > (table=chk2) user=Administrator (auth:SIMPLE) id=2
> state=ROLLEDBACK
> > > > > > exec-time=462msec
> > > > exception=org.apache.hadoop.hbase.TableExistsException:
> > > > > > chk2
> > > > > > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > > sessionid=0x1522613893e0004
> > > > > > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > > > > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > > > > > zookeeper.RecoverableZooKeeper: Process
> > > > identifier=hconnection-0x591287f8
> > > > > > connecting to ZooKeeper
> > > > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > > > 2016-01-09 18:40:29,482 INFO [thrift-worker-1]
> > zookeeper.ZooKeeper:
> > > > > > Initiating client connection,
> > > > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > > > > quorum=hostname:2181,hostname:2181,hostname:2181,
> baseZNode=/hbase
> > > > > > 2016-01-09 18:40:29,553 INFO
> > > > [thrift-worker-1-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
> > SASL
> > > > > > (unknown error)
> > > > > > 2016-01-09 18:40:29,559 INFO
> > > > [thrift-worker-1-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Socket connection established to
> > > > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > > > 2016-01-09 18:40:29,569 INFO
> > > > [thrift-worker-1-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008,
> negotiated
> > > > > timeout
> > > > > > = 40000
> > > > > > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > > sessionid=0x35226137e300008
> > > > > > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> > > > > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > > > > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > > 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> > > > > > zookeeper.RecoverableZooKeeper: Process
> > > > identifier=hconnection-0x6305dbeb
> > > > > > connecting to ZooKeeper
> > > > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > > > 2016-01-09 20:05:42,476 INFO [thrift-worker-13]
> > zookeeper.ZooKeeper:
> > > > > > Initiating client connection,
> > > > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > > > > quorum=hostname:2181,hostname:2181,hostname:2181,
> baseZNode=/hbase
> > > > > > 2016-01-09 20:05:42,507 INFO
> > > > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
> > SASL
> > > > > > (unknown error)
> > > > > > 2016-01-09 20:05:42,510 INFO
> > > > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Socket connection established to
> > > > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > > > 2016-01-09 20:05:42,708 INFO
> > > > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005,
> negotiated
> > > > > timeout
> > > > > > = 40000
> > > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > > sessionid=0x1522613893e0005
> > > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Rajeshkumar J
> > > > > >
> > > > > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Can you take a stack trace of the thrift server and pastebin
> the
> > > > trace
> > > > > ?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > > > > rajeshkumarit8292@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > >
> > > > > > > > I am connecting via Hbase thrift server to access records in
> > > Hbase
> > > > > and
> > > > > > I
> > > > > > > am
> > > > > > > > doing this in C# and i am using range scan to get records
> from
> > > the
> > > > > > table.
> > > > > > > > This seems to be working for few hours but after few hours it
> > > > didn't
> > > > > > > > returned any records and then i searched hbase logs i found
> > this
> > > > > > > >
> > > > > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > > > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > > > > zookeeper
> > > > > > > > sessionid=0x3520b1721650006
> > > > > > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
> > > > > zookeeper.ZooKeeper:
> > > > > > > > Session: 0x3520b1721650006 closed
> > > > > > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > > > >
> > > > > > > > And I am using hbase 0.98.13
> > > > > > > >
> > > > > > > > I have asked this question before and some user suggested me
> to
> > > > > upgrade
> > > > > > > to
> > > > > > > > new version so i have upgraded HBase to 1.1.2. But now it
> also
> > it
> > > > > > didn't
> > > > > > > > return any records but different log generated as shown below
> > > > > > > >
> > > > > > > > 2016-01-09 20:05:42,708 INFO
> > > > > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > > > > zookeeper.ClientCnxn: Session establishment complete on
> server
> > > > > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005,
> negotiated
> > > > > > timeout =
> > > > > > > > 40000
> > > > > > > > 2016-01-09 20:15:45,541 INFO
> [ConnectionCache_ChoreService_1]
> > > > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > > > zookeeper
> > > > > > > > sessionid=0x1522613893e0005
> > > > > > > > 2016-01-09 20:15:45,567 INFO
> [ConnectionCache_ChoreService_1]
> > > > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > > > >
> > > > > > > > Can any one help me in this?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Rajeshkumar J
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Hbase thrift server issues
Posted by Ted Yu <yu...@gmail.com>.
I took a look at the stack trace you posted.
Some thrift worker was in the state shown in HBASE-14926
I wonder if the fix for HBASE-14926 would help in your case.
Cheers
On Mon, Jan 11, 2016 at 5:11 AM, Rajeshkumar J <ra...@gmail.com>
wrote:
> Hi Ted,
>
> Plz ignore my previous mail as the log is taken when hbase is working
> fine. Please find the logs below when Hbase didn't return any records
>
> http://pastebin.com/APYjiGSP
>
> Thanks
>
> On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > When you search for 'java jstack tutorial' using google, you would find a
> > lot of tutorials.
> >
> > You can use 'ps aux' command to search for the following class:
> > org.apache.hadoop.hbase.thrift.ThriftServer
> >
> > Apply jstack on the corresponding process id.
> >
> > Cheers
> >
> > On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > > I don't know how how to take stacktrace using jstack. But i tried
> > using
> > > some commands but it fails. can you help me in this?
> > >
> > > Thanks
> > >
> > > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > What I meant was to capture stack trace of the thrift server (using
> > > jstack
> > > > command) when your query is running.
> > > >
> > > > This would give us some more clue as to the state thrift server was
> in
> > > when
> > > > the problem happened.
> > > >
> > > > Cheers
> > > >
> > > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > This is what I found in the Hbase thrift server logs
> > > > >
> > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:java.library.path=;C:\Hadoop\bin
> > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:java.compiler=<NA>
> > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:os.name=Windows Server 2012
> > > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:os.arch=amd64
> > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:os.version=6.2
> > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:user.name=xxxx
> > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:user.home=C:\
> > > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > Client
> > > > > environment:user.dir=C:\HBase\bin
> > > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0]
> zookeeper.ZooKeeper:
> > > > > Initiating client connection,
> > > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > > 2016-01-09 06:27:43,339 INFO
> > > [thrift-worker-0-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
> SASL
> > > > > (unknown error)
> > > > > 2016-01-09 06:27:43,342 INFO
> > > [thrift-worker-0-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Socket connection established to
> > > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > > 2016-01-09 06:27:43,365 INFO
> > > [thrift-worker-0-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > > > timeout
> > > > > = 40000
> > > > > 2016-01-09 07:14:10,607 INFO
> > > > > [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager
> Timeout
> > > > > Monitor missed its start time
> > > > > 2016-01-09 07:16:43,511 INFO
> > > > > [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > > DATA_BLOCK_ENCODING
> > > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS
> > =>
> > > > '1',
> > > > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > > > 'false',
> > > > > BLOCKCACHE => 'true'}
> > > > > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > > > > procedure2.ProcedureExecutor: Rolledback procedure
> > CreateTableProcedure
> > > > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > > > > exec-time=462msec
> > > exception=org.apache.hadoop.hbase.TableExistsException:
> > > > > chk2
> > > > > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > > > > client.ConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > > sessionid=0x1522613893e0004
> > > > > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > > > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > > > > zookeeper.RecoverableZooKeeper: Process
> > > identifier=hconnection-0x591287f8
> > > > > connecting to ZooKeeper
> > > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > > 2016-01-09 18:40:29,482 INFO [thrift-worker-1]
> zookeeper.ZooKeeper:
> > > > > Initiating client connection,
> > > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > > 2016-01-09 18:40:29,553 INFO
> > > [thrift-worker-1-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
> SASL
> > > > > (unknown error)
> > > > > 2016-01-09 18:40:29,559 INFO
> > > [thrift-worker-1-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Socket connection established to
> > > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > > 2016-01-09 18:40:29,569 INFO
> > > [thrift-worker-1-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> > > > timeout
> > > > > = 40000
> > > > > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> > > > > client.ConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > > sessionid=0x35226137e300008
> > > > > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> > > > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > > > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> > > > > zookeeper.RecoverableZooKeeper: Process
> > > identifier=hconnection-0x6305dbeb
> > > > > connecting to ZooKeeper
> > > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > > 2016-01-09 20:05:42,476 INFO [thrift-worker-13]
> zookeeper.ZooKeeper:
> > > > > Initiating client connection,
> > > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > > 2016-01-09 20:05:42,507 INFO
> > > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using
> SASL
> > > > > (unknown error)
> > > > > 2016-01-09 20:05:42,510 INFO
> > > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Socket connection established to
> > > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > > 2016-01-09 20:05:42,708 INFO
> > > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > > timeout
> > > > > = 40000
> > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > > client.ConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > > sessionid=0x1522613893e0005
> > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Rajeshkumar J
> > > > >
> > > > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >
> > > > > > Can you take a stack trace of the thrift server and pastebin the
> > > trace
> > > > ?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > > > rajeshkumarit8292@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > >
> > > > > > > I am connecting via Hbase thrift server to access records in
> > Hbase
> > > > and
> > > > > I
> > > > > > am
> > > > > > > doing this in C# and i am using range scan to get records from
> > the
> > > > > table.
> > > > > > > This seems to be working for few hours but after few hours it
> > > didn't
> > > > > > > returned any records and then i searched hbase logs i found
> this
> > > > > > >
> > > > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > > > zookeeper
> > > > > > > sessionid=0x3520b1721650006
> > > > > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
> > > > zookeeper.ZooKeeper:
> > > > > > > Session: 0x3520b1721650006 closed
> > > > > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > > >
> > > > > > > And I am using hbase 0.98.13
> > > > > > >
> > > > > > > I have asked this question before and some user suggested me to
> > > > upgrade
> > > > > > to
> > > > > > > new version so i have upgraded HBase to 1.1.2. But now it also
> it
> > > > > didn't
> > > > > > > return any records but different log generated as shown below
> > > > > > >
> > > > > > > 2016-01-09 20:05:42,708 INFO
> > > > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > > > timeout =
> > > > > > > 40000
> > > > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > > zookeeper
> > > > > > > sessionid=0x1522613893e0005
> > > > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > > >
> > > > > > > Can any one help me in this?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Rajeshkumar J
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Hbase thrift server issues
Posted by Rajeshkumar J <ra...@gmail.com>.
Hi Ted,
Plz ignore my previous mail as the log is taken when hbase is working
fine. Please find the logs below when Hbase didn't return any records
http://pastebin.com/APYjiGSP
Thanks
On Mon, Jan 11, 2016 at 11:19 AM, Ted Yu <yu...@gmail.com> wrote:
> When you search for 'java jstack tutorial' using google, you would find a
> lot of tutorials.
>
> You can use 'ps aux' command to search for the following class:
> org.apache.hadoop.hbase.thrift.ThriftServer
>
> Apply jstack on the corresponding process id.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > I don't know how how to take stacktrace using jstack. But i tried
> using
> > some commands but it fails. can you help me in this?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > What I meant was to capture stack trace of the thrift server (using
> > jstack
> > > command) when your query is running.
> > >
> > > This would give us some more clue as to the state thrift server was in
> > when
> > > the problem happened.
> > >
> > > Cheers
> > >
> > > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > This is what I found in the Hbase thrift server logs
> > > >
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.library.path=;C:\Hadoop\bin
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:java.compiler=<NA>
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.name=Windows Server 2012
> > > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.arch=amd64
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:os.version=6.2
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.name=xxxx
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.home=C:\
> > > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Client
> > > > environment:user.dir=C:\HBase\bin
> > > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 06:27:43,339 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 06:27:43,342 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 06:27:43,365 INFO
> > [thrift-worker-0-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 07:14:10,607 INFO
> > > > [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > > Monitor missed its start time
> > > > 2016-01-09 07:16:43,511 INFO
> > > > [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > > DATA_BLOCK_ENCODING
> > > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS
> =>
> > > '1',
> > > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > > 'false',
> > > > BLOCKCACHE => 'true'}
> > > > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > > > procedure2.ProcedureExecutor: Rolledback procedure
> CreateTableProcedure
> > > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > > > exec-time=462msec
> > exception=org.apache.hadoop.hbase.TableExistsException:
> > > > chk2
> > > > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0004
> > > > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > > > zookeeper.RecoverableZooKeeper: Process
> > identifier=hconnection-0x591287f8
> > > > connecting to ZooKeeper
> > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > 2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 18:40:29,553 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 18:40:29,559 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 18:40:29,569 INFO
> > [thrift-worker-1-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x35226137e300008
> > > > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > > 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> > > > zookeeper.RecoverableZooKeeper: Process
> > identifier=hconnection-0x6305dbeb
> > > > connecting to ZooKeeper
> > > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > > 2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper:
> > > > Initiating client connection,
> > > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > > 2016-01-09 20:05:42,507 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > > (unknown error)
> > > > 2016-01-09 20:05:42,510 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > hostname/xx.xx.x.xx:2181, initiating session
> > > > 2016-01-09 20:05:42,708 INFO
> > > [thrift-worker-13-SendThread(hostname:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > timeout
> > > > = 40000
> > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0005
> > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > >
> > > >
> > > > Thanks,
> > > > Rajeshkumar J
> > > >
> > > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >
> > > > > Can you take a stack trace of the thrift server and pastebin the
> > trace
> > > ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > > rajeshkumarit8292@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > I am connecting via Hbase thrift server to access records in
> Hbase
> > > and
> > > > I
> > > > > am
> > > > > > doing this in C# and i am using range scan to get records from
> the
> > > > table.
> > > > > > This seems to be working for few hours but after few hours it
> > didn't
> > > > > > returned any records and then i searched hbase logs i found this
> > > > > >
> > > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > > zookeeper
> > > > > > sessionid=0x3520b1721650006
> > > > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
> > > zookeeper.ZooKeeper:
> > > > > > Session: 0x3520b1721650006 closed
> > > > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > >
> > > > > > And I am using hbase 0.98.13
> > > > > >
> > > > > > I have asked this question before and some user suggested me to
> > > upgrade
> > > > > to
> > > > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > > > didn't
> > > > > > return any records but different log generated as shown below
> > > > > >
> > > > > > 2016-01-09 20:05:42,708 INFO
> > > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > > timeout =
> > > > > > 40000
> > > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > > > client.ConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > > sessionid=0x1522613893e0005
> > > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > > >
> > > > > > Can any one help me in this?
> > > > > >
> > > > > > Regards,
> > > > > > Rajeshkumar J
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Hbase thrift server issues
Posted by Ted Yu <yu...@gmail.com>.
When you search for 'java jstack tutorial' using google, you would find a
lot of tutorials.
You can use 'ps aux' command to search for the following class:
org.apache.hadoop.hbase.thrift.ThriftServer
Apply jstack on the corresponding process id.
Cheers
On Sun, Jan 10, 2016 at 9:43 PM, Rajeshkumar J <ra...@gmail.com>
wrote:
> Hi Ted,
>
> I don't know how how to take stacktrace using jstack. But i tried using
> some commands but it fails. can you help me in this?
>
> Thanks
>
> On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > What I meant was to capture stack trace of the thrift server (using
> jstack
> > command) when your query is running.
> >
> > This would give us some more clue as to the state thrift server was in
> when
> > the problem happened.
> >
> > Cheers
> >
> > On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > > This is what I found in the Hbase thrift server logs
> > >
> > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:java.library.path=;C:\Hadoop\bin
> > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:java.io.tmpdir=C:\Windows\TEMP\
> > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:java.compiler=<NA>
> > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:os.name=Windows Server 2012
> > > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:os.arch=amd64
> > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:os.version=6.2
> > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:user.name=xxxx
> > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:user.home=C:\
> > > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Client
> > > environment:user.dir=C:\HBase\bin
> > > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > > Initiating client connection,
> > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > 2016-01-09 06:27:43,339 INFO
> [thrift-worker-0-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > (unknown error)
> > > 2016-01-09 06:27:43,342 INFO
> [thrift-worker-0-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > hostname/xx.xx.x.xx:2181, initiating session
> > > 2016-01-09 06:27:43,365 INFO
> [thrift-worker-0-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> > timeout
> > > = 40000
> > > 2016-01-09 07:14:10,607 INFO
> > > [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > > Monitor missed its start time
> > > 2016-01-09 07:16:43,511 INFO
> > > [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > > Client=Administrator/null create 'chk2', {NAME => 'fa',
> > DATA_BLOCK_ENCODING
> > > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
> > '1',
> > > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> > 'false',
> > > BLOCKCACHE => 'true'}
> > > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > > procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> > > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > > exec-time=462msec
> exception=org.apache.hadoop.hbase.TableExistsException:
> > > chk2
> > > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x1522613893e0004
> > > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > > zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x591287f8
> > > connecting to ZooKeeper
> > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > 2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
> > > Initiating client connection,
> > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > 2016-01-09 18:40:29,553 INFO
> [thrift-worker-1-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > (unknown error)
> > > 2016-01-09 18:40:29,559 INFO
> [thrift-worker-1-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > hostname/xx.xx.x.xx:2181, initiating session
> > > 2016-01-09 18:40:29,569 INFO
> [thrift-worker-1-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> > timeout
> > > = 40000
> > > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x35226137e300008
> > > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > > 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> > > zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x6305dbeb
> > > connecting to ZooKeeper
> > ensemble=hostname:2181,hostname:2181,hostname:2181
> > > 2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper:
> > > Initiating client connection,
> > > connectString=hostname:2181,hostname:2181,hostname:2181
> > > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > > 2016-01-09 20:05:42,507 INFO
> > [thrift-worker-13-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > > (unknown error)
> > > 2016-01-09 20:05:42,510 INFO
> > [thrift-worker-13-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > hostname/xx.xx.x.xx:2181, initiating session
> > > 2016-01-09 20:05:42,708 INFO
> > [thrift-worker-13-SendThread(hostname:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > timeout
> > > = 40000
> > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x1522613893e0005
> > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > >
> > >
> > > Thanks,
> > > Rajeshkumar J
> > >
> > > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Can you take a stack trace of the thrift server and pastebin the
> trace
> > ?
> > > >
> > > > Thanks
> > > >
> > > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > > rajeshkumarit8292@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > I am connecting via Hbase thrift server to access records in Hbase
> > and
> > > I
> > > > am
> > > > > doing this in C# and i am using range scan to get records from the
> > > table.
> > > > > This seems to be working for few hours but after few hours it
> didn't
> > > > > returned any records and then i searched hbase logs i found this
> > > > >
> > > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > > > client.HConnectionManager$HConnectionImplementation: Closing
> > zookeeper
> > > > > sessionid=0x3520b1721650006
> > > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
> > zookeeper.ZooKeeper:
> > > > > Session: 0x3520b1721650006 closed
> > > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > >
> > > > > And I am using hbase 0.98.13
> > > > >
> > > > > I have asked this question before and some user suggested me to
> > upgrade
> > > > to
> > > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > > didn't
> > > > > return any records but different log generated as shown below
> > > > >
> > > > > 2016-01-09 20:05:42,708 INFO
> > [thrift-worker-13-SendThread(xxxx:2181)]
> > > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > > timeout =
> > > > > 40000
> > > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > > client.ConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > > sessionid=0x1522613893e0005
> > > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > > > zookeeper.ClientCnxn: EventThread shut down
> > > > >
> > > > > Can any one help me in this?
> > > > >
> > > > > Regards,
> > > > > Rajeshkumar J
> > > > >
> > > >
> > >
> >
>
Re: Hbase thrift server issues
Posted by Rajeshkumar J <ra...@gmail.com>.
Hi Ted,
I don't know how how to take stacktrace using jstack. But i tried using
some commands but it fails. can you help me in this?
Thanks
On Sun, Jan 10, 2016 at 11:10 PM, Ted Yu <yu...@gmail.com> wrote:
> What I meant was to capture stack trace of the thrift server (using jstack
> command) when your query is running.
>
> This would give us some more clue as to the state thrift server was in when
> the problem happened.
>
> Cheers
>
> On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > This is what I found in the Hbase thrift server logs
> >
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.library.path=;C:\Hadoop\bin
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.io.tmpdir=C:\Windows\TEMP\
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:java.compiler=<NA>
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.name=Windows Server 2012
> > 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.arch=amd64
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:os.version=6.2
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.name=xxxx
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.home=C:\
> > 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Client
> > environment:user.dir=C:\HBase\bin
> > 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 06:27:43,339 INFO [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 06:27:43,342 INFO [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 06:27:43,365 INFO [thrift-worker-0-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated
> timeout
> > = 40000
> > 2016-01-09 07:14:10,607 INFO
> > [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> > master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> > Monitor missed its start time
> > 2016-01-09 07:16:43,511 INFO
> > [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> > Client=Administrator/null create 'chk2', {NAME => 'fa',
> DATA_BLOCK_ENCODING
> > => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
> '1',
> > COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> > KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY =>
> 'false',
> > BLOCKCACHE => 'true'}
> > 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> > procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> > (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> > exec-time=462msec exception=org.apache.hadoop.hbase.TableExistsException:
> > chk2
> > 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x1522613893e0004
> > 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> > 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> > 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> > zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x591287f8
> > connecting to ZooKeeper
> ensemble=hostname:2181,hostname:2181,hostname:2181
> > 2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 18:40:29,553 INFO [thrift-worker-1-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 18:40:29,559 INFO [thrift-worker-1-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 18:40:29,569 INFO [thrift-worker-1-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated
> timeout
> > = 40000
> > 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x35226137e300008
> > 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> > 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> > 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> > zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6305dbeb
> > connecting to ZooKeeper
> ensemble=hostname:2181,hostname:2181,hostname:2181
> > 2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper:
> > Initiating client connection,
> > connectString=hostname:2181,hostname:2181,hostname:2181
> > sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> > quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> > 2016-01-09 20:05:42,507 INFO
> [thrift-worker-13-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> > (unknown error)
> > 2016-01-09 20:05:42,510 INFO
> [thrift-worker-13-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > hostname/xx.xx.x.xx:2181, initiating session
> > 2016-01-09 20:05:42,708 INFO
> [thrift-worker-13-SendThread(hostname:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> timeout
> > = 40000
> > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x1522613893e0005
> > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> >
> >
> > Thanks,
> > Rajeshkumar J
> >
> > On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Can you take a stack trace of the thrift server and pastebin the trace
> ?
> > >
> > > Thanks
> > >
> > > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > > rajeshkumarit8292@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > I am connecting via Hbase thrift server to access records in Hbase
> and
> > I
> > > am
> > > > doing this in C# and i am using range scan to get records from the
> > table.
> > > > This seems to be working for few hours but after few hours it didn't
> > > > returned any records and then i searched hbase logs i found this
> > > >
> > > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > > client.HConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > sessionid=0x3520b1721650006
> > > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner]
> zookeeper.ZooKeeper:
> > > > Session: 0x3520b1721650006 closed
> > > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > >
> > > > And I am using hbase 0.98.13
> > > >
> > > > I have asked this question before and some user suggested me to
> upgrade
> > > to
> > > > new version so i have upgraded HBase to 1.1.2. But now it also it
> > didn't
> > > > return any records but different log generated as shown below
> > > >
> > > > 2016-01-09 20:05:42,708 INFO
> [thrift-worker-13-SendThread(xxxx:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> > timeout =
> > > > 40000
> > > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > > sessionid=0x1522613893e0005
> > > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > > zookeeper.ClientCnxn: EventThread shut down
> > > >
> > > > Can any one help me in this?
> > > >
> > > > Regards,
> > > > Rajeshkumar J
> > > >
> > >
> >
>
Re: Hbase thrift server issues
Posted by Ted Yu <yu...@gmail.com>.
What I meant was to capture stack trace of the thrift server (using jstack
command) when your query is running.
This would give us some more clue as to the state thrift server was in when
the problem happened.
Cheers
On Sun, Jan 10, 2016 at 9:11 AM, Rajeshkumar J <ra...@gmail.com>
wrote:
> Hi Ted,
>
> This is what I found in the Hbase thrift server logs
>
> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:java.library.path=;C:\Hadoop\bin
> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=C:\Windows\TEMP\
> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:os.name=Windows Server 2012
> 2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:os.version=6.2
> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:user.name=xxxx
> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:user.home=C:\
> 2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
> environment:user.dir=C:\HBase\bin
> 2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
> Initiating client connection,
> connectString=hostname:2181,hostname:2181,hostname:2181
> sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
> quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> 2016-01-09 06:27:43,339 INFO [thrift-worker-0-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2016-01-09 06:27:43,342 INFO [thrift-worker-0-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> hostname/xx.xx.x.xx:2181, initiating session
> 2016-01-09 06:27:43,365 INFO [thrift-worker-0-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated timeout
> = 40000
> 2016-01-09 07:14:10,607 INFO
> [hostname,60000,1452337815006_splitLogManager__ChoreService_1]
> master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
> Monitor missed its start time
> 2016-01-09 07:16:43,511 INFO
> [B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
> Client=Administrator/null create 'chk2', {NAME => 'fa', DATA_BLOCK_ENCODING
> => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'}
> 2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
> procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
> (table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
> exec-time=462msec exception=org.apache.hadoop.hbase.TableExistsException:
> chk2
> 2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x1522613893e0004
> 2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
> 2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2016-01-09 18:40:29,469 INFO [thrift-worker-1]
> zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x591287f8
> connecting to ZooKeeper ensemble=hostname:2181,hostname:2181,hostname:2181
> 2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
> Initiating client connection,
> connectString=hostname:2181,hostname:2181,hostname:2181
> sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
> quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> 2016-01-09 18:40:29,553 INFO [thrift-worker-1-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2016-01-09 18:40:29,559 INFO [thrift-worker-1-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> hostname/xx.xx.x.xx:2181, initiating session
> 2016-01-09 18:40:29,569 INFO [thrift-worker-1-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated timeout
> = 40000
> 2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x35226137e300008
> 2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
> 2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
> 2016-01-09 20:05:42,469 INFO [thrift-worker-13]
> zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6305dbeb
> connecting to ZooKeeper ensemble=hostname:2181,hostname:2181,hostname:2181
> 2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper:
> Initiating client connection,
> connectString=hostname:2181,hostname:2181,hostname:2181
> sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
> quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
> 2016-01-09 20:05:42,507 INFO [thrift-worker-13-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2016-01-09 20:05:42,510 INFO [thrift-worker-13-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> hostname/xx.xx.x.xx:2181, initiating session
> 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(hostname:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout
> = 40000
> 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x1522613893e0005
> 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
>
>
> Thanks,
> Rajeshkumar J
>
> On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Can you take a stack trace of the thrift server and pastebin the trace ?
> >
> > Thanks
> >
> > On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> > rajeshkumarit8292@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > I am connecting via Hbase thrift server to access records in Hbase and
> I
> > am
> > > doing this in C# and i am using range scan to get records from the
> table.
> > > This seems to be working for few hours but after few hours it didn't
> > > returned any records and then i searched hbase logs i found this
> > >
> > > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x3520b1721650006
> > > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper:
> > > Session: 0x3520b1721650006 closed
> > > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > >
> > > And I am using hbase 0.98.13
> > >
> > > I have asked this question before and some user suggested me to upgrade
> > to
> > > new version so i have upgraded HBase to 1.1.2. But now it also it
> didn't
> > > return any records but different log generated as shown below
> > >
> > > 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(xxxx:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated
> timeout =
> > > 40000
> > > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x1522613893e0005
> > > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > > zookeeper.ClientCnxn: EventThread shut down
> > >
> > > Can any one help me in this?
> > >
> > > Regards,
> > > Rajeshkumar J
> > >
> >
>
Re: Hbase thrift server issues
Posted by Rajeshkumar J <ra...@gmail.com>.
Hi Ted,
This is what I found in the Hbase thrift server logs
2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:java.library.path=;C:\Hadoop\bin
2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=C:\Windows\TEMP\
2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:os.name=Windows Server 2012
2016-01-09 06:27:43,139 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:os.version=6.2
2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:user.name=xxxx
2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:user.home=C:\
2016-01-09 06:27:43,140 INFO [thrift-worker-0] zookeeper.ZooKeeper: Client
environment:user.dir=C:\HBase\bin
2016-01-09 06:27:43,142 INFO [thrift-worker-0] zookeeper.ZooKeeper:
Initiating client connection,
connectString=hostname:2181,hostname:2181,hostname:2181
sessionTimeout=90000 watcher=hconnection-0x45e35460x0,
quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
2016-01-09 06:27:43,339 INFO [thrift-worker-0-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
(unknown error)
2016-01-09 06:27:43,342 INFO [thrift-worker-0-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Socket connection established to
hostname/xx.xx.x.xx:2181, initiating session
2016-01-09 06:27:43,365 INFO [thrift-worker-0-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0004, negotiated timeout
= 40000
2016-01-09 07:14:10,607 INFO
[hostname,60000,1452337815006_splitLogManager__ChoreService_1]
master.SplitLogManager$TimeoutMonitor: Chore: SplitLogManager Timeout
Monitor missed its start time
2016-01-09 07:16:43,511 INFO
[B.defaultRpcServer.handler=8,queue=2,port=60000] master.HMaster:
Client=Administrator/null create 'chk2', {NAME => 'fa', DATA_BLOCK_ENCODING
=> 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}
2016-01-09 07:16:44,121 INFO [ProcedureExecutorThread-1]
procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure
(table=chk2) user=Administrator (auth:SIMPLE) id=2 state=ROLLEDBACK
exec-time=462msec exception=org.apache.hadoop.hbase.TableExistsException:
chk2
2016-01-09 07:49:55,574 INFO [ConnectionCache_ChoreService_1]
client.ConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x1522613893e0004
2016-01-09 07:49:55,629 INFO [ConnectionCache_ChoreService_1]
zookeeper.ZooKeeper: Session: 0x1522613893e0004 closed
2016-01-09 07:49:55,661 INFO [thrift-worker-0-EventThread]
zookeeper.ClientCnxn: EventThread shut down
2016-01-09 18:40:29,469 INFO [thrift-worker-1]
zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x591287f8
connecting to ZooKeeper ensemble=hostname:2181,hostname:2181,hostname:2181
2016-01-09 18:40:29,482 INFO [thrift-worker-1] zookeeper.ZooKeeper:
Initiating client connection,
connectString=hostname:2181,hostname:2181,hostname:2181
sessionTimeout=90000 watcher=hconnection-0x591287f80x0,
quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
2016-01-09 18:40:29,553 INFO [thrift-worker-1-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
(unknown error)
2016-01-09 18:40:29,559 INFO [thrift-worker-1-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Socket connection established to
hostname/xx.xx.x.xx:2181, initiating session
2016-01-09 18:40:29,569 INFO [thrift-worker-1-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
hostname/xx.xx.x.xx:2181, sessionid = 0x35226137e300008, negotiated timeout
= 40000
2016-01-09 18:54:25,558 INFO [ConnectionCache_ChoreService_1]
client.ConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x35226137e300008
2016-01-09 18:54:25,566 INFO [ConnectionCache_ChoreService_1]
zookeeper.ZooKeeper: Session: 0x35226137e300008 closed
2016-01-09 18:54:25,578 INFO [thrift-worker-1-EventThread]
zookeeper.ClientCnxn: EventThread shut down
2016-01-09 20:05:42,469 INFO [thrift-worker-13]
zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6305dbeb
connecting to ZooKeeper ensemble=hostname:2181,hostname:2181,hostname:2181
2016-01-09 20:05:42,476 INFO [thrift-worker-13] zookeeper.ZooKeeper:
Initiating client connection,
connectString=hostname:2181,hostname:2181,hostname:2181
sessionTimeout=90000 watcher=hconnection-0x6305dbeb0x0,
quorum=hostname:2181,hostname:2181,hostname:2181, baseZNode=/hbase
2016-01-09 20:05:42,507 INFO [thrift-worker-13-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
hostname/xx.xx.x.xx:2181. Will not attempt to authenticate using SASL
(unknown error)
2016-01-09 20:05:42,510 INFO [thrift-worker-13-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Socket connection established to
hostname/xx.xx.x.xx:2181, initiating session
2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(hostname:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
hostname/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout
= 40000
2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
client.ConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x1522613893e0005
2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
Thanks,
Rajeshkumar J
On Sun, Jan 10, 2016 at 10:31 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you take a stack trace of the thrift server and pastebin the trace ?
>
> Thanks
>
> On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi,
> >
> >
> > I am connecting via Hbase thrift server to access records in Hbase and I
> am
> > doing this in C# and i am using range scan to get records from the table.
> > This seems to be working for few hours but after few hours it didn't
> > returned any records and then i searched hbase logs i found this
> >
> > 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x3520b1721650006
> > 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper:
> > Session: 0x3520b1721650006 closed
> > 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> >
> > And I am using hbase 0.98.13
> >
> > I have asked this question before and some user suggested me to upgrade
> to
> > new version so i have upgraded HBase to 1.1.2. But now it also it didn't
> > return any records but different log generated as shown below
> >
> > 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(xxxx:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout =
> > 40000
> > 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> > client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x1522613893e0005
> > 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> > zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> > 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> > zookeeper.ClientCnxn: EventThread shut down
> >
> > Can any one help me in this?
> >
> > Regards,
> > Rajeshkumar J
> >
>
Re: Hbase thrift server issues
Posted by Ted Yu <yu...@gmail.com>.
Can you take a stack trace of the thrift server and pastebin the trace ?
Thanks
On Sun, Jan 10, 2016 at 8:56 AM, Rajeshkumar J <ra...@gmail.com>
wrote:
> Hi,
>
>
> I am connecting via Hbase thrift server to access records in Hbase and I am
> doing this in C# and i am using range scan to get records from the table.
> This seems to be working for few hours but after few hours it didn't
> returned any records and then i searched hbase logs i found this
>
> 2016-01-04 00:44:54,880 INFO [ConnectionCleaner]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x3520b1721650006
> 2016-01-04 00:44:54,891 INFO [ConnectionCleaner] zookeeper.ZooKeeper:
> Session: 0x3520b1721650006 closed
> 2016-01-04 00:44:54,891 INFO [thrift-worker-0-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
>
> And I am using hbase 0.98.13
>
> I have asked this question before and some user suggested me to upgrade to
> new version so i have upgraded HBase to 1.1.2. But now it also it didn't
> return any records but different log generated as shown below
>
> 2016-01-09 20:05:42,708 INFO [thrift-worker-13-SendThread(xxxx:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> xxxx/xx.xx.x.xx:2181, sessionid = 0x1522613893e0005, negotiated timeout =
> 40000
> 2016-01-09 20:15:45,541 INFO [ConnectionCache_ChoreService_1]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x1522613893e0005
> 2016-01-09 20:15:45,567 INFO [ConnectionCache_ChoreService_1]
> zookeeper.ZooKeeper: Session: 0x1522613893e0005 closed
> 2016-01-09 20:15:45,567 INFO [thrift-worker-13-EventThread]
> zookeeper.ClientCnxn: EventThread shut down
>
> Can any one help me in this?
>
> Regards,
> Rajeshkumar J
>