You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Serega Sheypak <se...@gmail.com> on 2014/12/12 11:47:51 UTC
HConnectionManager leaks with zookeeper conection oo many connections
from /my.tomcat.server.com - max is 60
Hi, I'm using HConnectionManager from java servlet
Looks like it's leaking, all my zookeepers complains that there is too many
connections from servlet hosts.
Typical line from lZK log:
oo many connections from /my.tomcat.server.com - max is 60
Here is code sample
public class BaseConnection {
private static final Logger LOG =
LoggerFactory.getLogger(BaseConnection.class);
protected void close(HConnection connection){
try{
if(connection == null){
return;
}
connection.close();
}
catch (Exception e){
LOG.warn("Error while closing HConnection", e);
}
}
protected void close(HTableInterface hTable){
try{
if(hTable == null){
return;
}
hTable.close();
}
catch (Exception e){
LOG.warn("Error while closing HTable", e);
}
}
}
sample PUT code from subclass:
public void put(List<MyBean> entries){
HConnection hConnection = null;
HTableInterface hTable = null;
try {
List<Put> puts = new ArrayList<Put>(entries.size());
for(MyBean myBean : entries){
puts.add(new MyBeanSerDe().createPut(myBean));
}
hConnection = HConnectionManager.createConnection(configuration);
hTable = hConnection.getTable(NAME_B);
hTable.put(puts);
}catch (Exception e){
LOG.error("Error while doing bulk put", e);
}finally{
close(hTable);
close(hConnection);
}
}
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
was: 200-300 ms per request
now: <80 ms request
request=full trip from servlet to HBase and back to response.
2014-12-15 22:40 GMT+03:00 lars hofhansl <la...@apache.org>:
>
> Excellent! Should be quite a bit faster too.
> -- Lars
> From: Serega Sheypak <se...@gmail.com>
> To: user <us...@hbase.apache.org>
> Cc: lars hofhansl <la...@apache.org>
> Sent: Monday, December 15, 2014 5:57 AM
> Subject: Re: HConnectionManager leaks with zookeeper conection oo many
> connections from /my.tomcat.server.com - max is 60
>
> Hi, the problem is gone.
> I did what you say :)
> Thanks!
>
>
>
> 2014-12-13 22:38 GMT+03:00 Serega Sheypak <se...@gmail.com>:
>
> > Great, I'll refactor the code. and report back
> >
> > 2014-12-13 22:36 GMT+03:00 Stack <st...@duboce.net>:
> >>
> >> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <
> >> serega.sheypak@gmail.com>
> >> wrote:
> >> >
> >> > So the idea is
> >> > 1. instantiate HConnection using HConnectionManager once
> >> > 2. Create HTable instance for each Servlet.doPost and close after
> >> operation
> >> > is done.
> >> > Is that correct?
> >> >
> >>
> >>
> >> Yes.
> >>
> >>
> >>
> >> >
> >> > Do region locations cached in this case?
> >> > Are ZK connections to region/ZK reused?
> >> >
> >>
> >> Yes
> >>
> >>
> >> > Can I "harm" put/get data because of Servlet concurrency? Each
> >> connection
> >> > could be used in many threads fro different HTable instances for the
> >> same
> >> > HBase table?
> >> >
> >> >
> >> Its a bug if the above has issues.
> >>
> >> St.Ack
> >>
> >>
> >>
> >> >
> >> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
> >> > >
> >> > > Note also that the createConnection part is somewhat expensive
> >> (creates a
> >> > > new thread pool for use with Puts, also does a ZK lookup, etc).If
> >> > possible
> >> > > create the connection ahead of time and only get/close an HTable per
> >> > > request/thread.
> >> > > -- Lars
> >> > > From: Serega Sheypak <se...@gmail.com>
> >> > > To: user <us...@hbase.apache.org>
> >> > > Sent: Friday, December 12, 2014 11:45 AM
> >> > > Subject: Re: HConnectionManager leaks with zookeeper conection oo
> >> many
> >> > > connections from /my.tomcat.server.com - max is 60
> >> > >
> >> > > i have 10K doPost/doGet requests per second.
> >> > > Servlet is NOT single-threaded. each doPost/doGet
> >> > > invokes these lines (encapsulated in DAO):
> >> > >
> >> > > 16 HConnection connection =
> >> > > HConnectionManager.createConnection(config);
> >> > > 17 HTableInterface table =
> >> > > connection.getTable(TableName.valueOf("table1"));
> >> > >
> >> > > and
> >> > >
> >> > > 24 } finally {
> >> > > 25 table.close();
> >> > > 26 connection.close();
> >> > > 27 }
> >> > >
> >> > > I assumed that this static construction
> >> > >
> >> > > 16 HConnection connection =
> >> > > HConnectionManager.createConnection(config);
> >> > >
> >> > > correctly handles multi-threaded access somewhere deep inside.
> >> > > Right now I don't understand what do I do wrong.
> >> > >
> >> > > Try to wrap each your request in Runnable to emulate multi-threaded
> >> > > pressure on ZK. Your code is linear and mine is not, it's
> concurrent.
> >> > >
> >> > > Thanks....
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> >> > > >
> >> > > > I cannot reproduce. I stood up a cdh5.2 server and then
> copy/pasted
> >> > your
> >> > > > code adding in a put for each cycle. I ran loop 1000 times and no
> >> > > > complaint from zk.
> >> > > >
> >> > > > Tell me more (Is servlet doing single-threaded model? A single
> >> > > > Configuration is being used or new ones are being created per
> >> servlet
> >> > > > invocation?
> >> > > >
> >> > > > Below is code and output.
> >> > > >
> >> > > > (For better perf, cache the connection)
> >> > > > St.Ack
> >> > > >
> >> > > >
> >> > > > 1 package org.apache.hadoop.hbase;
> >> > > > 2
> >> > > > 3 import java.io.IOException;
> >> > > > 4
> >> > > > 5 import org.apache.hadoop.conf.Configuration;
> >> > > > 6 import org.apache.hadoop.hbase.client.HConnection;
> >> > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> >> > > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> >> > > > 9 import org.apache.hadoop.hbase.client.Put;
> >> > > > 10 import org.apache.hadoop.hbase.util.Bytes;
> >> > > > 11
> >> > > > 12 public class TestConnnection {
> >> > > > 13 public static void main(String[] args) throws IOException {
> >> > > > 14 Configuration config = HBaseConfiguration.create();
> >> > > > 15 for (int i = 0; i < 1000; i++) {
> >> > > > 16 HConnection connection =
> >> > > > HConnectionManager.createConnection(config);
> >> > > > 17 HTableInterface table =
> >> > > > connection.getTable(TableName.valueOf("table1"));
> >> > > > 18 byte [] cf = Bytes.toBytes("t");
> >> > > > 19 try {
> >> > > > 20 byte [] bytes = Bytes.toBytes(i);
> >> > > > 21 Put p = new Put(bytes);
> >> > > > 22 p.add(cf, cf, bytes);
> >> > > > 23 table.put(p);
> >> > > > 24 } finally {
> >> > > > 25 table.close();
> >> > > > 26 connection.close();
> >> > > > 27 }
> >> > > > 28 System.out.println("i=" + i);
> >> > > > 29 }
> >> > > > 30 }
> >> > > > 31 }
> >> > > >
> >> > > >
> >> > > > ....
> >> > > > 2014-12-12 11:26:10,397 INFO [main]
> zookeeper.RecoverableZooKeeper:
> >> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> >> > > > ensemble=localhost:2181
> >> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper:
> Initiating
> >> > > client
> >> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> >> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181,
> >> baseZNode=/hbase
> >> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Opening socket connection to server
> localhost/
> >> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
> >> (unknown
> >> > > > error)
> >> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
> >> > > > 127.0.0.1:2181, initiating session
> >> > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Session establishment complete on server
> >> > localhost/
> >> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated
> timeout =
> >> > > 40000
> >> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
> >> > Finished
> >> > > > with small scan at {ENCODED => 1588230740, NAME =>
> 'hbase:meta,,1',
> >> > > > STARTKEY => '', ENDKEY => ''}
> >> > > > 2014-12-12 11:26:10,406 INFO [main]
> >> > > > client.HConnectionManager$HConnectionImplementation: Closing
> >> zookeeper
> >> > > > sessionid=0x14a3fe0664b083a
> >> > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> >> > > > 0x14a3fe0664b083a closed
> >> > > > 2014-12-12 11:26:10,407 INFO [main-EventThread]
> >> zookeeper.ClientCnxn:
> >> > > > EventThread shut down
> >> > > > i=997
> >> > > > 2014-12-12 11:26:10,511 INFO [main]
> zookeeper.RecoverableZooKeeper:
> >> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> >> > > > ensemble=localhost:2181
> >> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper:
> Initiating
> >> > > client
> >> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> >> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181,
> >> baseZNode=/hbase
> >> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Opening socket connection to server
> >> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate
> >> using
> >> > > SASL
> >> > > > (unknown error)
> >> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Socket connection established to
> >> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> >> > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Session establishment complete on server
> >> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
> >> > negotiated
> >> > > > timeout = 40000
> >> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
> >> > Finished
> >> > > > with small scan at {ENCODED => 1588230740, NAME =>
> 'hbase:meta,,1',
> >> > > > STARTKEY => '', ENDKEY => ''}
> >> > > > 2014-12-12 11:26:10,521 INFO [main]
> >> > > > client.HConnectionManager$HConnectionImplementation: Closing
> >> zookeeper
> >> > > > sessionid=0x14a3fe0664b083b
> >> > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> >> > > > 0x14a3fe0664b083b closed
> >> > > > 2014-12-12 11:26:10,521 INFO [main-EventThread]
> >> zookeeper.ClientCnxn:
> >> > > > EventThread shut down
> >> > > > i=998
> >> > > > 2014-12-12 11:26:10,627 INFO [main]
> zookeeper.RecoverableZooKeeper:
> >> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> >> > > > ensemble=localhost:2181
> >> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper:
> Initiating
> >> > > client
> >> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> >> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181,
> >> baseZNode=/hbase
> >> > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Opening socket connection to server
> localhost/
> >> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
> >> (unknown
> >> > > > error)
> >> > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
> >> > > > 127.0.0.1:2181, initiating session
> >> > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> >> > > > zookeeper.ClientCnxn: Session establishment complete on server
> >> > localhost/
> >> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated
> timeout =
> >> > > 40000
> >> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
> >> > Finished
> >> > > > with small scan at {ENCODED => 1588230740, NAME =>
> 'hbase:meta,,1',
> >> > > > STARTKEY => '', ENDKEY => ''}
> >> > > > 2014-12-12 11:26:10,638 INFO [main]
> >> > > > client.HConnectionManager$HConnectionImplementation: Closing
> >> zookeeper
> >> > > > sessionid=0x14a3fe0664b083c
> >> > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> >> > > > 0x14a3fe0664b083c closed
> >> > > > 2014-12-12 11:26:10,639 INFO [main-EventThread]
> >> zookeeper.ClientCnxn:
> >> > > > EventThread shut down
> >> > > > i=999
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> >> > > serega.sheypak@gmail.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi, I'm using CDH 5.2, 0.98
> >> > > > > I don't know how to use it correctly. I've just used this
> sample:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> >> > > > >
> >> > > > > HConnection connection =
> >> > HConnectionManager.createConnection(config);
> >> > > > > HTableInterface table =
> >> > > > connection.getTable(TableName.valueOf("table1"));
> >> > > > > try {
> >> > > > > // Use the table as needed, for a single operation and a
> single
> >> > > thread
> >> > > > > } finally {
> >> > > > > table.close();
> >> > > > > connection.close();
> >> > > > > }
> >> > > > >
> >> > > > >
> >> > > > > Functional testing OK, performance fails :((((
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> >> > > > > >
> >> > > > > > Does zk count go up on each request to the servlet? Which
> >> version
> >> > of
> >> > > > > hbase
> >> > > > > > so can try on this end? Do you have some client-side log?
> >> Better
> >> > if
> >> > > > you
> >> > > > > > cache the connection rather than make it each time since setup
> >> is
> >> > > > costly
> >> > > > > > but lets fix first problem first.
> >> > > > > > St.Ack
> >> > > > > >
> >> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> >> > > > > serega.sheypak@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi, I'm using HConnectionManager from java servlet
> >> > > > > > > Looks like it's leaking, all my zookeepers complains that
> >> there
> >> > is
> >> > > > too
> >> > > > > > many
> >> > > > > > > connections from servlet hosts.
> >> > > > > > > Typical line from lZK log:
> >> > > > > > >
> >> > > > > > > oo many connections from /my.tomcat.server.com - max is 60
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Here is code sample
> >> > > > > > >
> >> > > > > > > public class BaseConnection {
> >> > > > > > >
> >> > > > > > > private static final Logger LOG =
> >> > > > > > > LoggerFactory.getLogger(BaseConnection.class);
> >> > > > > > >
> >> > > > > > > protected void close(HConnection connection){
> >> > > > > > > try{
> >> > > > > > > if(connection == null){
> >> > > > > > > return;
> >> > > > > > > }
> >> > > > > > > connection.close();
> >> > > > > > > }
> >> > > > > > > catch (Exception e){
> >> > > > > > > LOG.warn("Error while closing HConnection", e);
> >> > > > > > > }
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > > protected void close(HTableInterface hTable){
> >> > > > > > > try{
> >> > > > > > > if(hTable == null){
> >> > > > > > > return;
> >> > > > > > > }
> >> > > > > > > hTable.close();
> >> > > > > > > }
> >> > > > > > > catch (Exception e){
> >> > > > > > > LOG.warn("Error while closing HTable", e);
> >> > > > > > > }
> >> > > > > > > }
> >> > > > > > > }
> >> > > > > > >
> >> > > > > > > sample PUT code from subclass:
> >> > > > > > >
> >> > > > > > > public void put(List<MyBean> entries){
> >> > > > > > > HConnection hConnection = null;
> >> > > > > > > HTableInterface hTable = null;
> >> > > > > > > try {
> >> > > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> >> > > > > > > for(MyBean myBean : entries){
> >> > > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> >> > > > > > > }
> >> > > > > > > hConnection =
> >> > > > > HConnectionManager.createConnection(configuration);
> >> > > > > > > hTable = hConnection.getTable(NAME_B);
> >> > > > > > > hTable.put(puts);
> >> > > > > > >
> >> > > > > > > }catch (Exception e){
> >> > > > > > > LOG.error("Error while doing bulk put", e);
> >> > > > > > > }finally{
> >> > > > > > > close(hTable);
> >> > > > > > > close(hConnection);
> >> > > > > > > }
> >> > > > > > > }
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
>
>
>
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by lars hofhansl <la...@apache.org>.
Excellent! Should be quite a bit faster too.
-- Lars
From: Serega Sheypak <se...@gmail.com>
To: user <us...@hbase.apache.org>
Cc: lars hofhansl <la...@apache.org>
Sent: Monday, December 15, 2014 5:57 AM
Subject: Re: HConnectionManager leaks with zookeeper conection oo many connections from /my.tomcat.server.com - max is 60
Hi, the problem is gone.
I did what you say :)
Thanks!
2014-12-13 22:38 GMT+03:00 Serega Sheypak <se...@gmail.com>:
> Great, I'll refactor the code. and report back
>
> 2014-12-13 22:36 GMT+03:00 Stack <st...@duboce.net>:
>>
>> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <
>> serega.sheypak@gmail.com>
>> wrote:
>> >
>> > So the idea is
>> > 1. instantiate HConnection using HConnectionManager once
>> > 2. Create HTable instance for each Servlet.doPost and close after
>> operation
>> > is done.
>> > Is that correct?
>> >
>>
>>
>> Yes.
>>
>>
>>
>> >
>> > Do region locations cached in this case?
>> > Are ZK connections to region/ZK reused?
>> >
>>
>> Yes
>>
>>
>> > Can I "harm" put/get data because of Servlet concurrency? Each
>> connection
>> > could be used in many threads fro different HTable instances for the
>> same
>> > HBase table?
>> >
>> >
>> Its a bug if the above has issues.
>>
>> St.Ack
>>
>>
>>
>> >
>> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
>> > >
>> > > Note also that the createConnection part is somewhat expensive
>> (creates a
>> > > new thread pool for use with Puts, also does a ZK lookup, etc).If
>> > possible
>> > > create the connection ahead of time and only get/close an HTable per
>> > > request/thread.
>> > > -- Lars
>> > > From: Serega Sheypak <se...@gmail.com>
>> > > To: user <us...@hbase.apache.org>
>> > > Sent: Friday, December 12, 2014 11:45 AM
>> > > Subject: Re: HConnectionManager leaks with zookeeper conection oo
>> many
>> > > connections from /my.tomcat.server.com - max is 60
>> > >
>> > > i have 10K doPost/doGet requests per second.
>> > > Servlet is NOT single-threaded. each doPost/doGet
>> > > invokes these lines (encapsulated in DAO):
>> > >
>> > > 16 HConnection connection =
>> > > HConnectionManager.createConnection(config);
>> > > 17 HTableInterface table =
>> > > connection.getTable(TableName.valueOf("table1"));
>> > >
>> > > and
>> > >
>> > > 24 } finally {
>> > > 25 table.close();
>> > > 26 connection.close();
>> > > 27 }
>> > >
>> > > I assumed that this static construction
>> > >
>> > > 16 HConnection connection =
>> > > HConnectionManager.createConnection(config);
>> > >
>> > > correctly handles multi-threaded access somewhere deep inside.
>> > > Right now I don't understand what do I do wrong.
>> > >
>> > > Try to wrap each your request in Runnable to emulate multi-threaded
>> > > pressure on ZK. Your code is linear and mine is not, it's concurrent.
>> > >
>> > > Thanks....
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
>> > > >
>> > > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
>> > your
>> > > > code adding in a put for each cycle. I ran loop 1000 times and no
>> > > > complaint from zk.
>> > > >
>> > > > Tell me more (Is servlet doing single-threaded model? A single
>> > > > Configuration is being used or new ones are being created per
>> servlet
>> > > > invocation?
>> > > >
>> > > > Below is code and output.
>> > > >
>> > > > (For better perf, cache the connection)
>> > > > St.Ack
>> > > >
>> > > >
>> > > > 1 package org.apache.hadoop.hbase;
>> > > > 2
>> > > > 3 import java.io.IOException;
>> > > > 4
>> > > > 5 import org.apache.hadoop.conf.Configuration;
>> > > > 6 import org.apache.hadoop.hbase.client.HConnection;
>> > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
>> > > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
>> > > > 9 import org.apache.hadoop.hbase.client.Put;
>> > > > 10 import org.apache.hadoop.hbase.util.Bytes;
>> > > > 11
>> > > > 12 public class TestConnnection {
>> > > > 13 public static void main(String[] args) throws IOException {
>> > > > 14 Configuration config = HBaseConfiguration.create();
>> > > > 15 for (int i = 0; i < 1000; i++) {
>> > > > 16 HConnection connection =
>> > > > HConnectionManager.createConnection(config);
>> > > > 17 HTableInterface table =
>> > > > connection.getTable(TableName.valueOf("table1"));
>> > > > 18 byte [] cf = Bytes.toBytes("t");
>> > > > 19 try {
>> > > > 20 byte [] bytes = Bytes.toBytes(i);
>> > > > 21 Put p = new Put(bytes);
>> > > > 22 p.add(cf, cf, bytes);
>> > > > 23 table.put(p);
>> > > > 24 } finally {
>> > > > 25 table.close();
>> > > > 26 connection.close();
>> > > > 27 }
>> > > > 28 System.out.println("i=" + i);
>> > > > 29 }
>> > > > 30 }
>> > > > 31 }
>> > > >
>> > > >
>> > > > ....
>> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
>> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
>> (unknown
>> > > > error)
>> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
>> > > > 127.0.0.1:2181, initiating session
>> > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > localhost/
>> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
>> > > 40000
>> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,406 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083a
>> > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083a closed
>> > > > 2014-12-12 11:26:10,407 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=997
>> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server
>> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate
>> using
>> > > SASL
>> > > > (unknown error)
>> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to
>> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
>> > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
>> > negotiated
>> > > > timeout = 40000
>> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,521 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083b
>> > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083b closed
>> > > > 2014-12-12 11:26:10,521 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=998
>> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
>> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
>> (unknown
>> > > > error)
>> > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
>> > > > 127.0.0.1:2181, initiating session
>> > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > localhost/
>> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
>> > > 40000
>> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,638 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083c
>> > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083c closed
>> > > > 2014-12-12 11:26:10,639 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=999
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
>> > > serega.sheypak@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Hi, I'm using CDH 5.2, 0.98
>> > > > > I don't know how to use it correctly. I've just used this sample:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>> > > > >
>> > > > > HConnection connection =
>> > HConnectionManager.createConnection(config);
>> > > > > HTableInterface table =
>> > > > connection.getTable(TableName.valueOf("table1"));
>> > > > > try {
>> > > > > // Use the table as needed, for a single operation and a single
>> > > thread
>> > > > > } finally {
>> > > > > table.close();
>> > > > > connection.close();
>> > > > > }
>> > > > >
>> > > > >
>> > > > > Functional testing OK, performance fails :((((
>> > > > >
>> > > > >
>> > > > >
>> > > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
>> > > > > >
>> > > > > > Does zk count go up on each request to the servlet? Which
>> version
>> > of
>> > > > > hbase
>> > > > > > so can try on this end? Do you have some client-side log?
>> Better
>> > if
>> > > > you
>> > > > > > cache the connection rather than make it each time since setup
>> is
>> > > > costly
>> > > > > > but lets fix first problem first.
>> > > > > > St.Ack
>> > > > > >
>> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
>> > > > > serega.sheypak@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi, I'm using HConnectionManager from java servlet
>> > > > > > > Looks like it's leaking, all my zookeepers complains that
>> there
>> > is
>> > > > too
>> > > > > > many
>> > > > > > > connections from servlet hosts.
>> > > > > > > Typical line from lZK log:
>> > > > > > >
>> > > > > > > oo many connections from /my.tomcat.server.com - max is 60
>> > > > > > >
>> > > > > > >
>> > > > > > > Here is code sample
>> > > > > > >
>> > > > > > > public class BaseConnection {
>> > > > > > >
>> > > > > > > private static final Logger LOG =
>> > > > > > > LoggerFactory.getLogger(BaseConnection.class);
>> > > > > > >
>> > > > > > > protected void close(HConnection connection){
>> > > > > > > try{
>> > > > > > > if(connection == null){
>> > > > > > > return;
>> > > > > > > }
>> > > > > > > connection.close();
>> > > > > > > }
>> > > > > > > catch (Exception e){
>> > > > > > > LOG.warn("Error while closing HConnection", e);
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > > > protected void close(HTableInterface hTable){
>> > > > > > > try{
>> > > > > > > if(hTable == null){
>> > > > > > > return;
>> > > > > > > }
>> > > > > > > hTable.close();
>> > > > > > > }
>> > > > > > > catch (Exception e){
>> > > > > > > LOG.warn("Error while closing HTable", e);
>> > > > > > > }
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > > > sample PUT code from subclass:
>> > > > > > >
>> > > > > > > public void put(List<MyBean> entries){
>> > > > > > > HConnection hConnection = null;
>> > > > > > > HTableInterface hTable = null;
>> > > > > > > try {
>> > > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
>> > > > > > > for(MyBean myBean : entries){
>> > > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
>> > > > > > > }
>> > > > > > > hConnection =
>> > > > > HConnectionManager.createConnection(configuration);
>> > > > > > > hTable = hConnection.getTable(NAME_B);
>> > > > > > > hTable.put(puts);
>> > > > > > >
>> > > > > > > }catch (Exception e){
>> > > > > > > LOG.error("Error while doing bulk put", e);
>> > > > > > > }finally{
>> > > > > > > close(hTable);
>> > > > > > > close(hConnection);
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
Hi, the problem is gone.
I did what you say :)
Thanks!
2014-12-13 22:38 GMT+03:00 Serega Sheypak <se...@gmail.com>:
> Great, I'll refactor the code. and report back
>
> 2014-12-13 22:36 GMT+03:00 Stack <st...@duboce.net>:
>>
>> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <
>> serega.sheypak@gmail.com>
>> wrote:
>> >
>> > So the idea is
>> > 1. instantiate HConnection using HConnectionManager once
>> > 2. Create HTable instance for each Servlet.doPost and close after
>> operation
>> > is done.
>> > Is that correct?
>> >
>>
>>
>> Yes.
>>
>>
>>
>> >
>> > Do region locations cached in this case?
>> > Are ZK connections to region/ZK reused?
>> >
>>
>> Yes
>>
>>
>> > Can I "harm" put/get data because of Servlet concurrency? Each
>> connection
>> > could be used in many threads fro different HTable instances for the
>> same
>> > HBase table?
>> >
>> >
>> Its a bug if the above has issues.
>>
>> St.Ack
>>
>>
>>
>> >
>> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
>> > >
>> > > Note also that the createConnection part is somewhat expensive
>> (creates a
>> > > new thread pool for use with Puts, also does a ZK lookup, etc).If
>> > possible
>> > > create the connection ahead of time and only get/close an HTable per
>> > > request/thread.
>> > > -- Lars
>> > > From: Serega Sheypak <se...@gmail.com>
>> > > To: user <us...@hbase.apache.org>
>> > > Sent: Friday, December 12, 2014 11:45 AM
>> > > Subject: Re: HConnectionManager leaks with zookeeper conection oo
>> many
>> > > connections from /my.tomcat.server.com - max is 60
>> > >
>> > > i have 10K doPost/doGet requests per second.
>> > > Servlet is NOT single-threaded. each doPost/doGet
>> > > invokes these lines (encapsulated in DAO):
>> > >
>> > > 16 HConnection connection =
>> > > HConnectionManager.createConnection(config);
>> > > 17 HTableInterface table =
>> > > connection.getTable(TableName.valueOf("table1"));
>> > >
>> > > and
>> > >
>> > > 24 } finally {
>> > > 25 table.close();
>> > > 26 connection.close();
>> > > 27 }
>> > >
>> > > I assumed that this static construction
>> > >
>> > > 16 HConnection connection =
>> > > HConnectionManager.createConnection(config);
>> > >
>> > > correctly handles multi-threaded access somewhere deep inside.
>> > > Right now I don't understand what do I do wrong.
>> > >
>> > > Try to wrap each your request in Runnable to emulate multi-threaded
>> > > pressure on ZK. Your code is linear and mine is not, it's concurrent.
>> > >
>> > > Thanks....
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
>> > > >
>> > > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
>> > your
>> > > > code adding in a put for each cycle. I ran loop 1000 times and no
>> > > > complaint from zk.
>> > > >
>> > > > Tell me more (Is servlet doing single-threaded model? A single
>> > > > Configuration is being used or new ones are being created per
>> servlet
>> > > > invocation?
>> > > >
>> > > > Below is code and output.
>> > > >
>> > > > (For better perf, cache the connection)
>> > > > St.Ack
>> > > >
>> > > >
>> > > > 1 package org.apache.hadoop.hbase;
>> > > > 2
>> > > > 3 import java.io.IOException;
>> > > > 4
>> > > > 5 import org.apache.hadoop.conf.Configuration;
>> > > > 6 import org.apache.hadoop.hbase.client.HConnection;
>> > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
>> > > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
>> > > > 9 import org.apache.hadoop.hbase.client.Put;
>> > > > 10 import org.apache.hadoop.hbase.util.Bytes;
>> > > > 11
>> > > > 12 public class TestConnnection {
>> > > > 13 public static void main(String[] args) throws IOException {
>> > > > 14 Configuration config = HBaseConfiguration.create();
>> > > > 15 for (int i = 0; i < 1000; i++) {
>> > > > 16 HConnection connection =
>> > > > HConnectionManager.createConnection(config);
>> > > > 17 HTableInterface table =
>> > > > connection.getTable(TableName.valueOf("table1"));
>> > > > 18 byte [] cf = Bytes.toBytes("t");
>> > > > 19 try {
>> > > > 20 byte [] bytes = Bytes.toBytes(i);
>> > > > 21 Put p = new Put(bytes);
>> > > > 22 p.add(cf, cf, bytes);
>> > > > 23 table.put(p);
>> > > > 24 } finally {
>> > > > 25 table.close();
>> > > > 26 connection.close();
>> > > > 27 }
>> > > > 28 System.out.println("i=" + i);
>> > > > 29 }
>> > > > 30 }
>> > > > 31 }
>> > > >
>> > > >
>> > > > ....
>> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
>> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
>> (unknown
>> > > > error)
>> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
>> > > > 127.0.0.1:2181, initiating session
>> > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > localhost/
>> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
>> > > 40000
>> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,406 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083a
>> > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083a closed
>> > > > 2014-12-12 11:26:10,407 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=997
>> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server
>> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate
>> using
>> > > SASL
>> > > > (unknown error)
>> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to
>> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
>> > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
>> > negotiated
>> > > > timeout = 40000
>> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,521 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083b
>> > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083b closed
>> > > > 2014-12-12 11:26:10,521 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=998
>> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
>> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
>> (unknown
>> > > > error)
>> > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
>> > > > 127.0.0.1:2181, initiating session
>> > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > localhost/
>> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
>> > > 40000
>> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,638 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083c
>> > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083c closed
>> > > > 2014-12-12 11:26:10,639 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=999
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
>> > > serega.sheypak@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Hi, I'm using CDH 5.2, 0.98
>> > > > > I don't know how to use it correctly. I've just used this sample:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>> > > > >
>> > > > > HConnection connection =
>> > HConnectionManager.createConnection(config);
>> > > > > HTableInterface table =
>> > > > connection.getTable(TableName.valueOf("table1"));
>> > > > > try {
>> > > > > // Use the table as needed, for a single operation and a single
>> > > thread
>> > > > > } finally {
>> > > > > table.close();
>> > > > > connection.close();
>> > > > > }
>> > > > >
>> > > > >
>> > > > > Functional testing OK, performance fails :((((
>> > > > >
>> > > > >
>> > > > >
>> > > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
>> > > > > >
>> > > > > > Does zk count go up on each request to the servlet? Which
>> version
>> > of
>> > > > > hbase
>> > > > > > so can try on this end? Do you have some client-side log?
>> Better
>> > if
>> > > > you
>> > > > > > cache the connection rather than make it each time since setup
>> is
>> > > > costly
>> > > > > > but lets fix first problem first.
>> > > > > > St.Ack
>> > > > > >
>> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
>> > > > > serega.sheypak@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi, I'm using HConnectionManager from java servlet
>> > > > > > > Looks like it's leaking, all my zookeepers complains that
>> there
>> > is
>> > > > too
>> > > > > > many
>> > > > > > > connections from servlet hosts.
>> > > > > > > Typical line from lZK log:
>> > > > > > >
>> > > > > > > oo many connections from /my.tomcat.server.com - max is 60
>> > > > > > >
>> > > > > > >
>> > > > > > > Here is code sample
>> > > > > > >
>> > > > > > > public class BaseConnection {
>> > > > > > >
>> > > > > > > private static final Logger LOG =
>> > > > > > > LoggerFactory.getLogger(BaseConnection.class);
>> > > > > > >
>> > > > > > > protected void close(HConnection connection){
>> > > > > > > try{
>> > > > > > > if(connection == null){
>> > > > > > > return;
>> > > > > > > }
>> > > > > > > connection.close();
>> > > > > > > }
>> > > > > > > catch (Exception e){
>> > > > > > > LOG.warn("Error while closing HConnection", e);
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > > > protected void close(HTableInterface hTable){
>> > > > > > > try{
>> > > > > > > if(hTable == null){
>> > > > > > > return;
>> > > > > > > }
>> > > > > > > hTable.close();
>> > > > > > > }
>> > > > > > > catch (Exception e){
>> > > > > > > LOG.warn("Error while closing HTable", e);
>> > > > > > > }
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > > > sample PUT code from subclass:
>> > > > > > >
>> > > > > > > public void put(List<MyBean> entries){
>> > > > > > > HConnection hConnection = null;
>> > > > > > > HTableInterface hTable = null;
>> > > > > > > try {
>> > > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
>> > > > > > > for(MyBean myBean : entries){
>> > > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
>> > > > > > > }
>> > > > > > > hConnection =
>> > > > > HConnectionManager.createConnection(configuration);
>> > > > > > > hTable = hConnection.getTable(NAME_B);
>> > > > > > > hTable.put(puts);
>> > > > > > >
>> > > > > > > }catch (Exception e){
>> > > > > > > LOG.error("Error while doing bulk put", e);
>> > > > > > > }finally{
>> > > > > > > close(hTable);
>> > > > > > > close(hConnection);
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
Great, I'll refactor the code. and report back
2014-12-13 22:36 GMT+03:00 Stack <st...@duboce.net>:
>
> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <serega.sheypak@gmail.com
> >
> wrote:
> >
> > So the idea is
> > 1. instantiate HConnection using HConnectionManager once
> > 2. Create HTable instance for each Servlet.doPost and close after
> operation
> > is done.
> > Is that correct?
> >
>
>
> Yes.
>
>
>
> >
> > Do region locations cached in this case?
> > Are ZK connections to region/ZK reused?
> >
>
> Yes
>
>
> > Can I "harm" put/get data because of Servlet concurrency? Each connection
> > could be used in many threads fro different HTable instances for the same
> > HBase table?
> >
> >
> Its a bug if the above has issues.
>
> St.Ack
>
>
>
> >
> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
> > >
> > > Note also that the createConnection part is somewhat expensive
> (creates a
> > > new thread pool for use with Puts, also does a ZK lookup, etc).If
> > possible
> > > create the connection ahead of time and only get/close an HTable per
> > > request/thread.
> > > -- Lars
> > > From: Serega Sheypak <se...@gmail.com>
> > > To: user <us...@hbase.apache.org>
> > > Sent: Friday, December 12, 2014 11:45 AM
> > > Subject: Re: HConnectionManager leaks with zookeeper conection oo many
> > > connections from /my.tomcat.server.com - max is 60
> > >
> > > i have 10K doPost/doGet requests per second.
> > > Servlet is NOT single-threaded. each doPost/doGet
> > > invokes these lines (encapsulated in DAO):
> > >
> > > 16 HConnection connection =
> > > HConnectionManager.createConnection(config);
> > > 17 HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > >
> > > and
> > >
> > > 24 } finally {
> > > 25 table.close();
> > > 26 connection.close();
> > > 27 }
> > >
> > > I assumed that this static construction
> > >
> > > 16 HConnection connection =
> > > HConnectionManager.createConnection(config);
> > >
> > > correctly handles multi-threaded access somewhere deep inside.
> > > Right now I don't understand what do I do wrong.
> > >
> > > Try to wrap each your request in Runnable to emulate multi-threaded
> > > pressure on ZK. Your code is linear and mine is not, it's concurrent.
> > >
> > > Thanks....
> > >
> > >
> > >
> > >
> > >
> > >
> > > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> > > >
> > > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
> > your
> > > > code adding in a put for each cycle. I ran loop 1000 times and no
> > > > complaint from zk.
> > > >
> > > > Tell me more (Is servlet doing single-threaded model? A single
> > > > Configuration is being used or new ones are being created per servlet
> > > > invocation?
> > > >
> > > > Below is code and output.
> > > >
> > > > (For better perf, cache the connection)
> > > > St.Ack
> > > >
> > > >
> > > > 1 package org.apache.hadoop.hbase;
> > > > 2
> > > > 3 import java.io.IOException;
> > > > 4
> > > > 5 import org.apache.hadoop.conf.Configuration;
> > > > 6 import org.apache.hadoop.hbase.client.HConnection;
> > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > > > 9 import org.apache.hadoop.hbase.client.Put;
> > > > 10 import org.apache.hadoop.hbase.util.Bytes;
> > > > 11
> > > > 12 public class TestConnnection {
> > > > 13 public static void main(String[] args) throws IOException {
> > > > 14 Configuration config = HBaseConfiguration.create();
> > > > 15 for (int i = 0; i < 1000; i++) {
> > > > 16 HConnection connection =
> > > > HConnectionManager.createConnection(config);
> > > > 17 HTableInterface table =
> > > > connection.getTable(TableName.valueOf("table1"));
> > > > 18 byte [] cf = Bytes.toBytes("t");
> > > > 19 try {
> > > > 20 byte [] bytes = Bytes.toBytes(i);
> > > > 21 Put p = new Put(bytes);
> > > > 22 p.add(cf, cf, bytes);
> > > > 23 table.put(p);
> > > > 24 } finally {
> > > > 25 table.close();
> > > > 26 connection.close();
> > > > 27 }
> > > > 28 System.out.println("i=" + i);
> > > > 29 }
> > > > 30 }
> > > > 31 }
> > > >
> > > >
> > > > ....
> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > > > ensemble=localhost:2181
> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> > > client
> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181,
> baseZNode=/hbase
> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > > error)
> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > > 127.0.0.1:2181, initiating session
> > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > localhost/
> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> > > 40000
> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
> > Finished
> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > > STARTKEY => '', ENDKEY => ''}
> > > > 2014-12-12 11:26:10,406 INFO [main]
> > > > client.HConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > sessionid=0x14a3fe0664b083a
> > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > > > 0x14a3fe0664b083a closed
> > > > 2014-12-12 11:26:10,407 INFO [main-EventThread]
> zookeeper.ClientCnxn:
> > > > EventThread shut down
> > > > i=997
> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > > > ensemble=localhost:2181
> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> > > client
> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181,
> baseZNode=/hbase
> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server
> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate
> using
> > > SASL
> > > > (unknown error)
> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to
> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
> > negotiated
> > > > timeout = 40000
> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
> > Finished
> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > > STARTKEY => '', ENDKEY => ''}
> > > > 2014-12-12 11:26:10,521 INFO [main]
> > > > client.HConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > sessionid=0x14a3fe0664b083b
> > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > > > 0x14a3fe0664b083b closed
> > > > 2014-12-12 11:26:10,521 INFO [main-EventThread]
> zookeeper.ClientCnxn:
> > > > EventThread shut down
> > > > i=998
> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > > > ensemble=localhost:2181
> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> > > client
> > > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181,
> baseZNode=/hbase
> > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > > error)
> > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > > 127.0.0.1:2181, initiating session
> > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > > > zookeeper.ClientCnxn: Session establishment complete on server
> > localhost/
> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> > > 40000
> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
> > Finished
> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > > STARTKEY => '', ENDKEY => ''}
> > > > 2014-12-12 11:26:10,638 INFO [main]
> > > > client.HConnectionManager$HConnectionImplementation: Closing
> zookeeper
> > > > sessionid=0x14a3fe0664b083c
> > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > > > 0x14a3fe0664b083c closed
> > > > 2014-12-12 11:26:10,639 INFO [main-EventThread]
> zookeeper.ClientCnxn:
> > > > EventThread shut down
> > > > i=999
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> > > serega.sheypak@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi, I'm using CDH 5.2, 0.98
> > > > > I don't know how to use it correctly. I've just used this sample:
> > > > >
> > > > >
> > > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > > > >
> > > > > HConnection connection =
> > HConnectionManager.createConnection(config);
> > > > > HTableInterface table =
> > > > connection.getTable(TableName.valueOf("table1"));
> > > > > try {
> > > > > // Use the table as needed, for a single operation and a single
> > > thread
> > > > > } finally {
> > > > > table.close();
> > > > > connection.close();
> > > > > }
> > > > >
> > > > >
> > > > > Functional testing OK, performance fails :((((
> > > > >
> > > > >
> > > > >
> > > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > > > > >
> > > > > > Does zk count go up on each request to the servlet? Which version
> > of
> > > > > hbase
> > > > > > so can try on this end? Do you have some client-side log? Better
> > if
> > > > you
> > > > > > cache the connection rather than make it each time since setup is
> > > > costly
> > > > > > but lets fix first problem first.
> > > > > > St.Ack
> > > > > >
> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > > > serega.sheypak@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > > > Looks like it's leaking, all my zookeepers complains that there
> > is
> > > > too
> > > > > > many
> > > > > > > connections from servlet hosts.
> > > > > > > Typical line from lZK log:
> > > > > > >
> > > > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > > > >
> > > > > > >
> > > > > > > Here is code sample
> > > > > > >
> > > > > > > public class BaseConnection {
> > > > > > >
> > > > > > > private static final Logger LOG =
> > > > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > > > >
> > > > > > > protected void close(HConnection connection){
> > > > > > > try{
> > > > > > > if(connection == null){
> > > > > > > return;
> > > > > > > }
> > > > > > > connection.close();
> > > > > > > }
> > > > > > > catch (Exception e){
> > > > > > > LOG.warn("Error while closing HConnection", e);
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > protected void close(HTableInterface hTable){
> > > > > > > try{
> > > > > > > if(hTable == null){
> > > > > > > return;
> > > > > > > }
> > > > > > > hTable.close();
> > > > > > > }
> > > > > > > catch (Exception e){
> > > > > > > LOG.warn("Error while closing HTable", e);
> > > > > > > }
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > > sample PUT code from subclass:
> > > > > > >
> > > > > > > public void put(List<MyBean> entries){
> > > > > > > HConnection hConnection = null;
> > > > > > > HTableInterface hTable = null;
> > > > > > > try {
> > > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > > > for(MyBean myBean : entries){
> > > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > > > }
> > > > > > > hConnection =
> > > > > HConnectionManager.createConnection(configuration);
> > > > > > > hTable = hConnection.getTable(NAME_B);
> > > > > > > hTable.put(puts);
> > > > > > >
> > > > > > > }catch (Exception e){
> > > > > > > LOG.error("Error while doing bulk put", e);
> > > > > > > }finally{
> > > > > > > close(hTable);
> > > > > > > close(hConnection);
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Stack <st...@duboce.net>.
On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <se...@gmail.com>
wrote:
>
> So the idea is
> 1. instantiate HConnection using HConnectionManager once
> 2. Create HTable instance for each Servlet.doPost and close after operation
> is done.
> Is that correct?
>
Yes.
>
> Do region locations cached in this case?
> Are ZK connections to region/ZK reused?
>
Yes
> Can I "harm" put/get data because of Servlet concurrency? Each connection
> could be used in many threads fro different HTable instances for the same
> HBase table?
>
>
Its a bug if the above has issues.
St.Ack
>
> 2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
> >
> > Note also that the createConnection part is somewhat expensive (creates a
> > new thread pool for use with Puts, also does a ZK lookup, etc).If
> possible
> > create the connection ahead of time and only get/close an HTable per
> > request/thread.
> > -- Lars
> > From: Serega Sheypak <se...@gmail.com>
> > To: user <us...@hbase.apache.org>
> > Sent: Friday, December 12, 2014 11:45 AM
> > Subject: Re: HConnectionManager leaks with zookeeper conection oo many
> > connections from /my.tomcat.server.com - max is 60
> >
> > i have 10K doPost/doGet requests per second.
> > Servlet is NOT single-threaded. each doPost/doGet
> > invokes these lines (encapsulated in DAO):
> >
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> > 17 HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> >
> > and
> >
> > 24 } finally {
> > 25 table.close();
> > 26 connection.close();
> > 27 }
> >
> > I assumed that this static construction
> >
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> >
> > correctly handles multi-threaded access somewhere deep inside.
> > Right now I don't understand what do I do wrong.
> >
> > Try to wrap each your request in Runnable to emulate multi-threaded
> > pressure on ZK. Your code is linear and mine is not, it's concurrent.
> >
> > Thanks....
> >
> >
> >
> >
> >
> >
> > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> > >
> > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
> your
> > > code adding in a put for each cycle. I ran loop 1000 times and no
> > > complaint from zk.
> > >
> > > Tell me more (Is servlet doing single-threaded model? A single
> > > Configuration is being used or new ones are being created per servlet
> > > invocation?
> > >
> > > Below is code and output.
> > >
> > > (For better perf, cache the connection)
> > > St.Ack
> > >
> > >
> > > 1 package org.apache.hadoop.hbase;
> > > 2
> > > 3 import java.io.IOException;
> > > 4
> > > 5 import org.apache.hadoop.conf.Configuration;
> > > 6 import org.apache.hadoop.hbase.client.HConnection;
> > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > > 9 import org.apache.hadoop.hbase.client.Put;
> > > 10 import org.apache.hadoop.hbase.util.Bytes;
> > > 11
> > > 12 public class TestConnnection {
> > > 13 public static void main(String[] args) throws IOException {
> > > 14 Configuration config = HBaseConfiguration.create();
> > > 15 for (int i = 0; i < 1000; i++) {
> > > 16 HConnection connection =
> > > HConnectionManager.createConnection(config);
> > > 17 HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > > 18 byte [] cf = Bytes.toBytes("t");
> > > 19 try {
> > > 20 byte [] bytes = Bytes.toBytes(i);
> > > 21 Put p = new Put(bytes);
> > > 22 p.add(cf, cf, bytes);
> > > 23 table.put(p);
> > > 24 } finally {
> > > 25 table.close();
> > > 26 connection.close();
> > > 27 }
> > > 28 System.out.println("i=" + i);
> > > 29 }
> > > 30 }
> > > 31 }
> > >
> > >
> > > ....
> > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > error)
> > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > 127.0.0.1:2181, initiating session
> > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> localhost/
> > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> > 40000
> > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,406 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083a
> > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083a closed
> > > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=997
> > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> > SASL
> > > (unknown error)
> > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
> negotiated
> > > timeout = 40000
> > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,521 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083b
> > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083b closed
> > > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=998
> > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > error)
> > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > 127.0.0.1:2181, initiating session
> > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> localhost/
> > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> > 40000
> > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,638 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083c
> > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083c closed
> > > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=999
> > >
> > >
> > >
> > >
> > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> > serega.sheypak@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi, I'm using CDH 5.2, 0.98
> > > > I don't know how to use it correctly. I've just used this sample:
> > > >
> > > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > > >
> > > > HConnection connection =
> HConnectionManager.createConnection(config);
> > > > HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > > > try {
> > > > // Use the table as needed, for a single operation and a single
> > thread
> > > > } finally {
> > > > table.close();
> > > > connection.close();
> > > > }
> > > >
> > > >
> > > > Functional testing OK, performance fails :((((
> > > >
> > > >
> > > >
> > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > > > >
> > > > > Does zk count go up on each request to the servlet? Which version
> of
> > > > hbase
> > > > > so can try on this end? Do you have some client-side log? Better
> if
> > > you
> > > > > cache the connection rather than make it each time since setup is
> > > costly
> > > > > but lets fix first problem first.
> > > > > St.Ack
> > > > >
> > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > > serega.sheypak@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > > Looks like it's leaking, all my zookeepers complains that there
> is
> > > too
> > > > > many
> > > > > > connections from servlet hosts.
> > > > > > Typical line from lZK log:
> > > > > >
> > > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > > >
> > > > > >
> > > > > > Here is code sample
> > > > > >
> > > > > > public class BaseConnection {
> > > > > >
> > > > > > private static final Logger LOG =
> > > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > > >
> > > > > > protected void close(HConnection connection){
> > > > > > try{
> > > > > > if(connection == null){
> > > > > > return;
> > > > > > }
> > > > > > connection.close();
> > > > > > }
> > > > > > catch (Exception e){
> > > > > > LOG.warn("Error while closing HConnection", e);
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > protected void close(HTableInterface hTable){
> > > > > > try{
> > > > > > if(hTable == null){
> > > > > > return;
> > > > > > }
> > > > > > hTable.close();
> > > > > > }
> > > > > > catch (Exception e){
> > > > > > LOG.warn("Error while closing HTable", e);
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > sample PUT code from subclass:
> > > > > >
> > > > > > public void put(List<MyBean> entries){
> > > > > > HConnection hConnection = null;
> > > > > > HTableInterface hTable = null;
> > > > > > try {
> > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > > for(MyBean myBean : entries){
> > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > > }
> > > > > > hConnection =
> > > > HConnectionManager.createConnection(configuration);
> > > > > > hTable = hConnection.getTable(NAME_B);
> > > > > > hTable.put(puts);
> > > > > >
> > > > > > }catch (Exception e){
> > > > > > LOG.error("Error while doing bulk put", e);
> > > > > > }finally{
> > > > > > close(hTable);
> > > > > > close(hConnection);
> > > > > > }
> > > > > > }
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
So the idea is
1. instantiate HConnection using HConnectionManager once
2. Create HTable instance for each Servlet.doPost and close after operation
is done.
Is that correct?
Do region locations cached in this case?
Are ZK connections to region/ZK reused?
Can I "harm" put/get data because of Servlet concurrency? Each connection
could be used in many threads fro different HTable instances for the same
HBase table?
2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>:
>
> Note also that the createConnection part is somewhat expensive (creates a
> new thread pool for use with Puts, also does a ZK lookup, etc).If possible
> create the connection ahead of time and only get/close an HTable per
> request/thread.
> -- Lars
> From: Serega Sheypak <se...@gmail.com>
> To: user <us...@hbase.apache.org>
> Sent: Friday, December 12, 2014 11:45 AM
> Subject: Re: HConnectionManager leaks with zookeeper conection oo many
> connections from /my.tomcat.server.com - max is 60
>
> i have 10K doPost/doGet requests per second.
> Servlet is NOT single-threaded. each doPost/doGet
> invokes these lines (encapsulated in DAO):
>
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
> 17 HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
>
> and
>
> 24 } finally {
> 25 table.close();
> 26 connection.close();
> 27 }
>
> I assumed that this static construction
>
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
>
> correctly handles multi-threaded access somewhere deep inside.
> Right now I don't understand what do I do wrong.
>
> Try to wrap each your request in Runnable to emulate multi-threaded
> pressure on ZK. Your code is linear and mine is not, it's concurrent.
>
> Thanks....
>
>
>
>
>
>
> 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> >
> > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
> > code adding in a put for each cycle. I ran loop 1000 times and no
> > complaint from zk.
> >
> > Tell me more (Is servlet doing single-threaded model? A single
> > Configuration is being used or new ones are being created per servlet
> > invocation?
> >
> > Below is code and output.
> >
> > (For better perf, cache the connection)
> > St.Ack
> >
> >
> > 1 package org.apache.hadoop.hbase;
> > 2
> > 3 import java.io.IOException;
> > 4
> > 5 import org.apache.hadoop.conf.Configuration;
> > 6 import org.apache.hadoop.hbase.client.HConnection;
> > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > 9 import org.apache.hadoop.hbase.client.Put;
> > 10 import org.apache.hadoop.hbase.util.Bytes;
> > 11
> > 12 public class TestConnnection {
> > 13 public static void main(String[] args) throws IOException {
> > 14 Configuration config = HBaseConfiguration.create();
> > 15 for (int i = 0; i < 1000; i++) {
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> > 17 HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > 18 byte [] cf = Bytes.toBytes("t");
> > 19 try {
> > 20 byte [] bytes = Bytes.toBytes(i);
> > 21 Put p = new Put(bytes);
> > 22 p.add(cf, cf, bytes);
> > 23 table.put(p);
> > 24 } finally {
> > 25 table.close();
> > 26 connection.close();
> > 27 }
> > 28 System.out.println("i=" + i);
> > 29 }
> > 30 }
> > 31 }
> >
> >
> > ....
> > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,406 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083a
> > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083a closed
> > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=997
> > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> SASL
> > (unknown error)
> > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
> > timeout = 40000
> > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,521 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083b
> > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083b closed
> > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=998
> > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,638 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083c
> > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083c closed
> > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=999
> >
> >
> >
> >
> > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> serega.sheypak@gmail.com
> > >
> > wrote:
> >
> > > Hi, I'm using CDH 5.2, 0.98
> > > I don't know how to use it correctly. I've just used this sample:
> > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > >
> > > HConnection connection = HConnectionManager.createConnection(config);
> > > HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > > try {
> > > // Use the table as needed, for a single operation and a single
> thread
> > > } finally {
> > > table.close();
> > > connection.close();
> > > }
> > >
> > >
> > > Functional testing OK, performance fails :((((
> > >
> > >
> > >
> > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > > >
> > > > Does zk count go up on each request to the servlet? Which version of
> > > hbase
> > > > so can try on this end? Do you have some client-side log? Better if
> > you
> > > > cache the connection rather than make it each time since setup is
> > costly
> > > > but lets fix first problem first.
> > > > St.Ack
> > > >
> > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > serega.sheypak@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > Looks like it's leaking, all my zookeepers complains that there is
> > too
> > > > many
> > > > > connections from servlet hosts.
> > > > > Typical line from lZK log:
> > > > >
> > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > >
> > > > >
> > > > > Here is code sample
> > > > >
> > > > > public class BaseConnection {
> > > > >
> > > > > private static final Logger LOG =
> > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > >
> > > > > protected void close(HConnection connection){
> > > > > try{
> > > > > if(connection == null){
> > > > > return;
> > > > > }
> > > > > connection.close();
> > > > > }
> > > > > catch (Exception e){
> > > > > LOG.warn("Error while closing HConnection", e);
> > > > > }
> > > > > }
> > > > >
> > > > > protected void close(HTableInterface hTable){
> > > > > try{
> > > > > if(hTable == null){
> > > > > return;
> > > > > }
> > > > > hTable.close();
> > > > > }
> > > > > catch (Exception e){
> > > > > LOG.warn("Error while closing HTable", e);
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > sample PUT code from subclass:
> > > > >
> > > > > public void put(List<MyBean> entries){
> > > > > HConnection hConnection = null;
> > > > > HTableInterface hTable = null;
> > > > > try {
> > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > for(MyBean myBean : entries){
> > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > }
> > > > > hConnection =
> > > HConnectionManager.createConnection(configuration);
> > > > > hTable = hConnection.getTable(NAME_B);
> > > > > hTable.put(puts);
> > > > >
> > > > > }catch (Exception e){
> > > > > LOG.error("Error while doing bulk put", e);
> > > > > }finally{
> > > > > close(hTable);
> > > > > close(hConnection);
> > > > > }
> > > > > }
> > > > >
> > > >
> > >
> >
>
>
>
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by lars hofhansl <la...@apache.org>.
Note also that the createConnection part is somewhat expensive (creates a new thread pool for use with Puts, also does a ZK lookup, etc).If possible create the connection ahead of time and only get/close an HTable per request/thread.
-- Lars
From: Serega Sheypak <se...@gmail.com>
To: user <us...@hbase.apache.org>
Sent: Friday, December 12, 2014 11:45 AM
Subject: Re: HConnectionManager leaks with zookeeper conection oo many connections from /my.tomcat.server.com - max is 60
i have 10K doPost/doGet requests per second.
Servlet is NOT single-threaded. each doPost/doGet
invokes these lines (encapsulated in DAO):
16 HConnection connection =
HConnectionManager.createConnection(config);
17 HTableInterface table =
connection.getTable(TableName.valueOf("table1"));
and
24 } finally {
25 table.close();
26 connection.close();
27 }
I assumed that this static construction
16 HConnection connection =
HConnectionManager.createConnection(config);
correctly handles multi-threaded access somewhere deep inside.
Right now I don't understand what do I do wrong.
Try to wrap each your request in Runnable to emulate multi-threaded
pressure on ZK. Your code is linear and mine is not, it's concurrent.
Thanks....
2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
>
> I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
> code adding in a put for each cycle. I ran loop 1000 times and no
> complaint from zk.
>
> Tell me more (Is servlet doing single-threaded model? A single
> Configuration is being used or new ones are being created per servlet
> invocation?
>
> Below is code and output.
>
> (For better perf, cache the connection)
> St.Ack
>
>
> 1 package org.apache.hadoop.hbase;
> 2
> 3 import java.io.IOException;
> 4
> 5 import org.apache.hadoop.conf.Configuration;
> 6 import org.apache.hadoop.hbase.client.HConnection;
> 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> 8 import org.apache.hadoop.hbase.client.HTableInterface;
> 9 import org.apache.hadoop.hbase.client.Put;
> 10 import org.apache.hadoop.hbase.util.Bytes;
> 11
> 12 public class TestConnnection {
> 13 public static void main(String[] args) throws IOException {
> 14 Configuration config = HBaseConfiguration.create();
> 15 for (int i = 0; i < 1000; i++) {
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
> 17 HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
> 18 byte [] cf = Bytes.toBytes("t");
> 19 try {
> 20 byte [] bytes = Bytes.toBytes(i);
> 21 Put p = new Put(bytes);
> 22 p.add(cf, cf, bytes);
> 23 table.put(p);
> 24 } finally {
> 25 table.close();
> 26 connection.close();
> 27 }
> 28 System.out.println("i=" + i);
> 29 }
> 30 }
> 31 }
>
>
> ....
> 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> ensemble=localhost:2181
> 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server localhost/
> 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Socket connection established to localhost/
> 127.0.0.1:2181, initiating session
> 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server localhost/
> 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout = 40000
> 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
> with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> STARTKEY => '', ENDKEY => ''}
> 2014-12-12 11:26:10,406 INFO [main]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x14a3fe0664b083a
> 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> 0x14a3fe0664b083a closed
> 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> i=997
> 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> ensemble=localhost:2181
> 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> localhost/0:0:0:0:0:0:0:1:2181, initiating session
> 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
> timeout = 40000
> 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
> with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> STARTKEY => '', ENDKEY => ''}
> 2014-12-12 11:26:10,521 INFO [main]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x14a3fe0664b083b
> 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> 0x14a3fe0664b083b closed
> 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> i=998
> 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> ensemble=localhost:2181
> 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server localhost/
> 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Socket connection established to localhost/
> 127.0.0.1:2181, initiating session
> 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server localhost/
> 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout = 40000
> 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
> with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> STARTKEY => '', ENDKEY => ''}
> 2014-12-12 11:26:10,638 INFO [main]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x14a3fe0664b083c
> 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> 0x14a3fe0664b083c closed
> 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> i=999
>
>
>
>
> On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <serega.sheypak@gmail.com
> >
> wrote:
>
> > Hi, I'm using CDH 5.2, 0.98
> > I don't know how to use it correctly. I've just used this sample:
> >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> >
> > HConnection connection = HConnectionManager.createConnection(config);
> > HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
> > try {
> > // Use the table as needed, for a single operation and a single thread
> > } finally {
> > table.close();
> > connection.close();
> > }
> >
> >
> > Functional testing OK, performance fails :((((
> >
> >
> >
> > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > >
> > > Does zk count go up on each request to the servlet? Which version of
> > hbase
> > > so can try on this end? Do you have some client-side log? Better if
> you
> > > cache the connection rather than make it each time since setup is
> costly
> > > but lets fix first problem first.
> > > St.Ack
> > >
> > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > serega.sheypak@gmail.com>
> > > wrote:
> > >
> > > > Hi, I'm using HConnectionManager from java servlet
> > > > Looks like it's leaking, all my zookeepers complains that there is
> too
> > > many
> > > > connections from servlet hosts.
> > > > Typical line from lZK log:
> > > >
> > > > oo many connections from /my.tomcat.server.com - max is 60
> > > >
> > > >
> > > > Here is code sample
> > > >
> > > > public class BaseConnection {
> > > >
> > > > private static final Logger LOG =
> > > > LoggerFactory.getLogger(BaseConnection.class);
> > > >
> > > > protected void close(HConnection connection){
> > > > try{
> > > > if(connection == null){
> > > > return;
> > > > }
> > > > connection.close();
> > > > }
> > > > catch (Exception e){
> > > > LOG.warn("Error while closing HConnection", e);
> > > > }
> > > > }
> > > >
> > > > protected void close(HTableInterface hTable){
> > > > try{
> > > > if(hTable == null){
> > > > return;
> > > > }
> > > > hTable.close();
> > > > }
> > > > catch (Exception e){
> > > > LOG.warn("Error while closing HTable", e);
> > > > }
> > > > }
> > > > }
> > > >
> > > > sample PUT code from subclass:
> > > >
> > > > public void put(List<MyBean> entries){
> > > > HConnection hConnection = null;
> > > > HTableInterface hTable = null;
> > > > try {
> > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > for(MyBean myBean : entries){
> > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > }
> > > > hConnection =
> > HConnectionManager.createConnection(configuration);
> > > > hTable = hConnection.getTable(NAME_B);
> > > > hTable.put(puts);
> > > >
> > > > }catch (Exception e){
> > > > LOG.error("Error while doing bulk put", e);
> > > > }finally{
> > > > close(hTable);
> > > > close(hConnection);
> > > > }
> > > > }
> > > >
> > >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
Hm... confusing,
So here is the code path:
1. Servlet has doPost method
2. HistoryController instantiated during each doPost request.
HistoryController is Logic wrapper around low-level HBase persistence stuff
Each doPost invokes:
final HistoryController getHistoryController(){
return new HistoryController(configuration: getHBaseConfiguration())
}
getHistoryController().updateHistory(historyEntryVisitorId)
3. HistoryController has method:
private HistoryEntryConnection getConnection(){
new HistoryEntryConnection(configuration)
}
so each
updateHistory(historyEntryVisitorId)
gets new low-level HistoryEntryConnection
instance.
HistoryEntryConnection is dead simple, I've shown it previously
public void put(List<MyBean> entries){
HConnection hConnection = null;
HTableInterface hTable = null;
try {
List<Put> puts = new ArrayList<Put>(entries.size());
for(MyBean myBean : entries){
puts.add(new MyBeanSerDe().createPut(myBean));
}
hConnection = HConnectionManager.createConnection(configuration);
hTable = hConnection.getTable(NAME_B);
hTable.put(puts);
}catch (Exception e){
LOG.error("Error while doing bulk put", e);
}finally{
close(hTable);
close(hConnection);
}
}
Manager, Connection, HTable usage copy-pasted from javadoc. Nothing new.
BTW, is there any possibility to limit quantity of opened connections in
HConnectionManager internals ?
I have several web-apps using HBase as backends. Maybe their cumulative
pressure makes ZK go down...
2014-12-13 4:03 GMT+03:00 Stack <st...@duboce.net>:
>
> On Fri, Dec 12, 2014 at 11:45 AM, Serega Sheypak <serega.sheypak@gmail.com
> >
> wrote:
> >
> > i have 10K doPost/doGet requests per second.
> >
>
> How many concurrent threads going on in your tomcat? Is the Connection
> shared amongst all threads? Perhaps you have > 60 concurrent connections
> running at a time and each Connection has its own zk instance. Thread
> dump?
>
> Post all the code?
>
> I tried multithreaded with 100 threads but still can't repro (see code
> below). Next would be to get zk to dump concurrent connection count but
> lets see what your answers above are first.
>
> St.Ack
>
>
> public class TestConnnection {
>
> static class Putter extends Thread {
>
> Configuration c;
>
> Putter(final Configuration c, final String i) {
>
> super(i);
>
> this.c = c;
>
> }
>
>
> @Override
>
> public void run() {
>
> try {
>
> for (int i = 0; i < 100; i++) {
>
> HConnection connection = HConnectionManager.createConnection(this.c
> );
>
> HTableInterface table = connection.getTable(TableName.valueOf(
> "table1"));
>
> byte [] cf = Bytes.toBytes("t");
>
> try {
>
> byte [] bytes = Bytes.toBytes(i);
>
> Put p = new Put(bytes);
>
> p.add(cf, cf, bytes);
>
> table.put(p);
>
> } finally {
>
> table.close();
>
> connection.close();
>
> }
>
> System.out.println(getName() + ", i=" + i);
>
> }
>
> } catch (Throwable t) {
>
> throw new RuntimeException(t);
>
> }
>
> }
>
> };
>
>
> public static void main(String[] args) throws IOException {
>
> Configuration config = HBaseConfiguration.create();
>
> for (int i = 0; i < 100; i++) {
>
> Thread t = new Putter(config, "" + i);
>
> t.start();
>
> }
>
> }
>
> }
>
>
>
>
>
> > Servlet is NOT single-threaded. each doPost/doGet
> > invokes these lines (encapsulated in DAO):
> >
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> > 17 HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> >
> > and
> >
> > 24 } finally {
> > 25 table.close();
> > 26 connection.close();
> > 27 }
> >
> > I assumed that this static construction
> >
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> >
> > correctly handles multi-threaded access somewhere deep inside.
> > Right now I don't understand what do I do wrong.
> >
> > Try to wrap each your request in Runnable to emulate multi-threaded
> > pressure on ZK. Your code is linear and mine is not, it's concurrent.
> >
> > Thanks....
> >
> >
> >
> >
> > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> > >
> > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
> your
> > > code adding in a put for each cycle. I ran loop 1000 times and no
> > > complaint from zk.
> > >
> > > Tell me more (Is servlet doing single-threaded model? A single
> > > Configuration is being used or new ones are being created per servlet
> > > invocation?
> > >
> > > Below is code and output.
> > >
> > > (For better perf, cache the connection)
> > > St.Ack
> > >
> > >
> > > 1 package org.apache.hadoop.hbase;
> > > 2
> > > 3 import java.io.IOException;
> > > 4
> > > 5 import org.apache.hadoop.conf.Configuration;
> > > 6 import org.apache.hadoop.hbase.client.HConnection;
> > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > > 9 import org.apache.hadoop.hbase.client.Put;
> > > 10 import org.apache.hadoop.hbase.util.Bytes;
> > > 11
> > > 12 public class TestConnnection {
> > > 13 public static void main(String[] args) throws IOException {
> > > 14 Configuration config = HBaseConfiguration.create();
> > > 15 for (int i = 0; i < 1000; i++) {
> > > 16 HConnection connection =
> > > HConnectionManager.createConnection(config);
> > > 17 HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > > 18 byte [] cf = Bytes.toBytes("t");
> > > 19 try {
> > > 20 byte [] bytes = Bytes.toBytes(i);
> > > 21 Put p = new Put(bytes);
> > > 22 p.add(cf, cf, bytes);
> > > 23 table.put(p);
> > > 24 } finally {
> > > 25 table.close();
> > > 26 connection.close();
> > > 27 }
> > > 28 System.out.println("i=" + i);
> > > 29 }
> > > 30 }
> > > 31 }
> > >
> > >
> > > ....
> > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > error)
> > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > 127.0.0.1:2181, initiating session
> > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> localhost/
> > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> > 40000
> > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,406 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083a
> > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083a closed
> > > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=997
> > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> > SASL
> > > (unknown error)
> > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
> negotiated
> > > timeout = 40000
> > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,521 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083b
> > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083b closed
> > > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=998
> > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > error)
> > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > 127.0.0.1:2181, initiating session
> > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> localhost/
> > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> > 40000
> > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,638 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083c
> > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083c closed
> > > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=999
> > >
> > >
> > >
> > >
> > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> > serega.sheypak@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi, I'm using CDH 5.2, 0.98
> > > > I don't know how to use it correctly. I've just used this sample:
> > > >
> > > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > > >
> > > > HConnection connection =
> HConnectionManager.createConnection(config);
> > > > HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > > > try {
> > > > // Use the table as needed, for a single operation and a single
> > thread
> > > > } finally {
> > > > table.close();
> > > > connection.close();
> > > > }
> > > >
> > > >
> > > > Functional testing OK, performance fails :((((
> > > >
> > > >
> > > >
> > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > > > >
> > > > > Does zk count go up on each request to the servlet? Which version
> of
> > > > hbase
> > > > > so can try on this end? Do you have some client-side log? Better
> if
> > > you
> > > > > cache the connection rather than make it each time since setup is
> > > costly
> > > > > but lets fix first problem first.
> > > > > St.Ack
> > > > >
> > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > > serega.sheypak@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > > Looks like it's leaking, all my zookeepers complains that there
> is
> > > too
> > > > > many
> > > > > > connections from servlet hosts.
> > > > > > Typical line from lZK log:
> > > > > >
> > > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > > >
> > > > > >
> > > > > > Here is code sample
> > > > > >
> > > > > > public class BaseConnection {
> > > > > >
> > > > > > private static final Logger LOG =
> > > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > > >
> > > > > > protected void close(HConnection connection){
> > > > > > try{
> > > > > > if(connection == null){
> > > > > > return;
> > > > > > }
> > > > > > connection.close();
> > > > > > }
> > > > > > catch (Exception e){
> > > > > > LOG.warn("Error while closing HConnection", e);
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > protected void close(HTableInterface hTable){
> > > > > > try{
> > > > > > if(hTable == null){
> > > > > > return;
> > > > > > }
> > > > > > hTable.close();
> > > > > > }
> > > > > > catch (Exception e){
> > > > > > LOG.warn("Error while closing HTable", e);
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > sample PUT code from subclass:
> > > > > >
> > > > > > public void put(List<MyBean> entries){
> > > > > > HConnection hConnection = null;
> > > > > > HTableInterface hTable = null;
> > > > > > try {
> > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > > for(MyBean myBean : entries){
> > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > > }
> > > > > > hConnection =
> > > > HConnectionManager.createConnection(configuration);
> > > > > > hTable = hConnection.getTable(NAME_B);
> > > > > > hTable.put(puts);
> > > > > >
> > > > > > }catch (Exception e){
> > > > > > LOG.error("Error while doing bulk put", e);
> > > > > > }finally{
> > > > > > close(hTable);
> > > > > > close(hConnection);
> > > > > > }
> > > > > > }
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Stack <st...@duboce.net>.
On Fri, Dec 12, 2014 at 11:45 AM, Serega Sheypak <se...@gmail.com>
wrote:
>
> i have 10K doPost/doGet requests per second.
>
How many concurrent threads going on in your tomcat? Is the Connection
shared amongst all threads? Perhaps you have > 60 concurrent connections
running at a time and each Connection has its own zk instance. Thread dump?
Post all the code?
I tried multithreaded with 100 threads but still can't repro (see code
below). Next would be to get zk to dump concurrent connection count but
lets see what your answers above are first.
St.Ack
public class TestConnnection {
static class Putter extends Thread {
Configuration c;
Putter(final Configuration c, final String i) {
super(i);
this.c = c;
}
@Override
public void run() {
try {
for (int i = 0; i < 100; i++) {
HConnection connection = HConnectionManager.createConnection(this.c
);
HTableInterface table = connection.getTable(TableName.valueOf(
"table1"));
byte [] cf = Bytes.toBytes("t");
try {
byte [] bytes = Bytes.toBytes(i);
Put p = new Put(bytes);
p.add(cf, cf, bytes);
table.put(p);
} finally {
table.close();
connection.close();
}
System.out.println(getName() + ", i=" + i);
}
} catch (Throwable t) {
throw new RuntimeException(t);
}
}
};
public static void main(String[] args) throws IOException {
Configuration config = HBaseConfiguration.create();
for (int i = 0; i < 100; i++) {
Thread t = new Putter(config, "" + i);
t.start();
}
}
}
> Servlet is NOT single-threaded. each doPost/doGet
> invokes these lines (encapsulated in DAO):
>
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
> 17 HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
>
> and
>
> 24 } finally {
> 25 table.close();
> 26 connection.close();
> 27 }
>
> I assumed that this static construction
>
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
>
> correctly handles multi-threaded access somewhere deep inside.
> Right now I don't understand what do I do wrong.
>
> Try to wrap each your request in Runnable to emulate multi-threaded
> pressure on ZK. Your code is linear and mine is not, it's concurrent.
>
> Thanks....
>
>
>
>
> 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
> >
> > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
> > code adding in a put for each cycle. I ran loop 1000 times and no
> > complaint from zk.
> >
> > Tell me more (Is servlet doing single-threaded model? A single
> > Configuration is being used or new ones are being created per servlet
> > invocation?
> >
> > Below is code and output.
> >
> > (For better perf, cache the connection)
> > St.Ack
> >
> >
> > 1 package org.apache.hadoop.hbase;
> > 2
> > 3 import java.io.IOException;
> > 4
> > 5 import org.apache.hadoop.conf.Configuration;
> > 6 import org.apache.hadoop.hbase.client.HConnection;
> > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > 9 import org.apache.hadoop.hbase.client.Put;
> > 10 import org.apache.hadoop.hbase.util.Bytes;
> > 11
> > 12 public class TestConnnection {
> > 13 public static void main(String[] args) throws IOException {
> > 14 Configuration config = HBaseConfiguration.create();
> > 15 for (int i = 0; i < 1000; i++) {
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> > 17 HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > 18 byte [] cf = Bytes.toBytes("t");
> > 19 try {
> > 20 byte [] bytes = Bytes.toBytes(i);
> > 21 Put p = new Put(bytes);
> > 22 p.add(cf, cf, bytes);
> > 23 table.put(p);
> > 24 } finally {
> > 25 table.close();
> > 26 connection.close();
> > 27 }
> > 28 System.out.println("i=" + i);
> > 29 }
> > 30 }
> > 31 }
> >
> >
> > ....
> > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,406 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083a
> > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083a closed
> > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=997
> > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server
> > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> SASL
> > (unknown error)
> > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to
> > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server
> > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
> > timeout = 40000
> > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,521 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083b
> > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083b closed
> > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=998
> > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > ensemble=localhost:2181
> > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=localhost:2181 sessionTimeout=90000
> > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > error)
> > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Socket connection established to localhost/
> > 127.0.0.1:2181, initiating session
> > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server localhost/
> > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> 40000
> > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-12-12 11:26:10,638 INFO [main]
> > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > sessionid=0x14a3fe0664b083c
> > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > 0x14a3fe0664b083c closed
> > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> > EventThread shut down
> > i=999
> >
> >
> >
> >
> > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> serega.sheypak@gmail.com
> > >
> > wrote:
> >
> > > Hi, I'm using CDH 5.2, 0.98
> > > I don't know how to use it correctly. I've just used this sample:
> > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > >
> > > HConnection connection = HConnectionManager.createConnection(config);
> > > HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> > > try {
> > > // Use the table as needed, for a single operation and a single
> thread
> > > } finally {
> > > table.close();
> > > connection.close();
> > > }
> > >
> > >
> > > Functional testing OK, performance fails :((((
> > >
> > >
> > >
> > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > > >
> > > > Does zk count go up on each request to the servlet? Which version of
> > > hbase
> > > > so can try on this end? Do you have some client-side log? Better if
> > you
> > > > cache the connection rather than make it each time since setup is
> > costly
> > > > but lets fix first problem first.
> > > > St.Ack
> > > >
> > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > serega.sheypak@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > Looks like it's leaking, all my zookeepers complains that there is
> > too
> > > > many
> > > > > connections from servlet hosts.
> > > > > Typical line from lZK log:
> > > > >
> > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > >
> > > > >
> > > > > Here is code sample
> > > > >
> > > > > public class BaseConnection {
> > > > >
> > > > > private static final Logger LOG =
> > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > >
> > > > > protected void close(HConnection connection){
> > > > > try{
> > > > > if(connection == null){
> > > > > return;
> > > > > }
> > > > > connection.close();
> > > > > }
> > > > > catch (Exception e){
> > > > > LOG.warn("Error while closing HConnection", e);
> > > > > }
> > > > > }
> > > > >
> > > > > protected void close(HTableInterface hTable){
> > > > > try{
> > > > > if(hTable == null){
> > > > > return;
> > > > > }
> > > > > hTable.close();
> > > > > }
> > > > > catch (Exception e){
> > > > > LOG.warn("Error while closing HTable", e);
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > sample PUT code from subclass:
> > > > >
> > > > > public void put(List<MyBean> entries){
> > > > > HConnection hConnection = null;
> > > > > HTableInterface hTable = null;
> > > > > try {
> > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > for(MyBean myBean : entries){
> > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > }
> > > > > hConnection =
> > > HConnectionManager.createConnection(configuration);
> > > > > hTable = hConnection.getTable(NAME_B);
> > > > > hTable.put(puts);
> > > > >
> > > > > }catch (Exception e){
> > > > > LOG.error("Error while doing bulk put", e);
> > > > > }finally{
> > > > > close(hTable);
> > > > > close(hConnection);
> > > > > }
> > > > > }
> > > > >
> > > >
> > >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
i have 10K doPost/doGet requests per second.
Servlet is NOT single-threaded. each doPost/doGet
invokes these lines (encapsulated in DAO):
16 HConnection connection =
HConnectionManager.createConnection(config);
17 HTableInterface table =
connection.getTable(TableName.valueOf("table1"));
and
24 } finally {
25 table.close();
26 connection.close();
27 }
I assumed that this static construction
16 HConnection connection =
HConnectionManager.createConnection(config);
correctly handles multi-threaded access somewhere deep inside.
Right now I don't understand what do I do wrong.
Try to wrap each your request in Runnable to emulate multi-threaded
pressure on ZK. Your code is linear and mine is not, it's concurrent.
Thanks....
2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>:
>
> I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
> code adding in a put for each cycle. I ran loop 1000 times and no
> complaint from zk.
>
> Tell me more (Is servlet doing single-threaded model? A single
> Configuration is being used or new ones are being created per servlet
> invocation?
>
> Below is code and output.
>
> (For better perf, cache the connection)
> St.Ack
>
>
> 1 package org.apache.hadoop.hbase;
> 2
> 3 import java.io.IOException;
> 4
> 5 import org.apache.hadoop.conf.Configuration;
> 6 import org.apache.hadoop.hbase.client.HConnection;
> 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> 8 import org.apache.hadoop.hbase.client.HTableInterface;
> 9 import org.apache.hadoop.hbase.client.Put;
> 10 import org.apache.hadoop.hbase.util.Bytes;
> 11
> 12 public class TestConnnection {
> 13 public static void main(String[] args) throws IOException {
> 14 Configuration config = HBaseConfiguration.create();
> 15 for (int i = 0; i < 1000; i++) {
> 16 HConnection connection =
> HConnectionManager.createConnection(config);
> 17 HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
> 18 byte [] cf = Bytes.toBytes("t");
> 19 try {
> 20 byte [] bytes = Bytes.toBytes(i);
> 21 Put p = new Put(bytes);
> 22 p.add(cf, cf, bytes);
> 23 table.put(p);
> 24 } finally {
> 25 table.close();
> 26 connection.close();
> 27 }
> 28 System.out.println("i=" + i);
> 29 }
> 30 }
> 31 }
>
>
> ....
> 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> ensemble=localhost:2181
> 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server localhost/
> 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Socket connection established to localhost/
> 127.0.0.1:2181, initiating session
> 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server localhost/
> 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout = 40000
> 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
> with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> STARTKEY => '', ENDKEY => ''}
> 2014-12-12 11:26:10,406 INFO [main]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x14a3fe0664b083a
> 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> 0x14a3fe0664b083a closed
> 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> i=997
> 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> ensemble=localhost:2181
> 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server
> localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
> (unknown error)
> 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Socket connection established to
> localhost/0:0:0:0:0:0:0:1:2181, initiating session
> 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server
> localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
> timeout = 40000
> 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
> with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> STARTKEY => '', ENDKEY => ''}
> 2014-12-12 11:26:10,521 INFO [main]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x14a3fe0664b083b
> 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> 0x14a3fe0664b083b closed
> 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> i=998
> 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> ensemble=localhost:2181
> 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=localhost:2181 sessionTimeout=90000
> watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server localhost/
> 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Socket connection established to localhost/
> 127.0.0.1:2181, initiating session
> 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server localhost/
> 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout = 40000
> 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
> with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> STARTKEY => '', ENDKEY => ''}
> 2014-12-12 11:26:10,638 INFO [main]
> client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x14a3fe0664b083c
> 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> 0x14a3fe0664b083c closed
> 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> i=999
>
>
>
>
> On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <serega.sheypak@gmail.com
> >
> wrote:
>
> > Hi, I'm using CDH 5.2, 0.98
> > I don't know how to use it correctly. I've just used this sample:
> >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> >
> > HConnection connection = HConnectionManager.createConnection(config);
> > HTableInterface table =
> connection.getTable(TableName.valueOf("table1"));
> > try {
> > // Use the table as needed, for a single operation and a single thread
> > } finally {
> > table.close();
> > connection.close();
> > }
> >
> >
> > Functional testing OK, performance fails :((((
> >
> >
> >
> > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> > >
> > > Does zk count go up on each request to the servlet? Which version of
> > hbase
> > > so can try on this end? Do you have some client-side log? Better if
> you
> > > cache the connection rather than make it each time since setup is
> costly
> > > but lets fix first problem first.
> > > St.Ack
> > >
> > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > serega.sheypak@gmail.com>
> > > wrote:
> > >
> > > > Hi, I'm using HConnectionManager from java servlet
> > > > Looks like it's leaking, all my zookeepers complains that there is
> too
> > > many
> > > > connections from servlet hosts.
> > > > Typical line from lZK log:
> > > >
> > > > oo many connections from /my.tomcat.server.com - max is 60
> > > >
> > > >
> > > > Here is code sample
> > > >
> > > > public class BaseConnection {
> > > >
> > > > private static final Logger LOG =
> > > > LoggerFactory.getLogger(BaseConnection.class);
> > > >
> > > > protected void close(HConnection connection){
> > > > try{
> > > > if(connection == null){
> > > > return;
> > > > }
> > > > connection.close();
> > > > }
> > > > catch (Exception e){
> > > > LOG.warn("Error while closing HConnection", e);
> > > > }
> > > > }
> > > >
> > > > protected void close(HTableInterface hTable){
> > > > try{
> > > > if(hTable == null){
> > > > return;
> > > > }
> > > > hTable.close();
> > > > }
> > > > catch (Exception e){
> > > > LOG.warn("Error while closing HTable", e);
> > > > }
> > > > }
> > > > }
> > > >
> > > > sample PUT code from subclass:
> > > >
> > > > public void put(List<MyBean> entries){
> > > > HConnection hConnection = null;
> > > > HTableInterface hTable = null;
> > > > try {
> > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > for(MyBean myBean : entries){
> > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > }
> > > > hConnection =
> > HConnectionManager.createConnection(configuration);
> > > > hTable = hConnection.getTable(NAME_B);
> > > > hTable.put(puts);
> > > >
> > > > }catch (Exception e){
> > > > LOG.error("Error while doing bulk put", e);
> > > > }finally{
> > > > close(hTable);
> > > > close(hConnection);
> > > > }
> > > > }
> > > >
> > >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Stack <st...@duboce.net>.
I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted your
code adding in a put for each cycle. I ran loop 1000 times and no
complaint from zk.
Tell me more (Is servlet doing single-threaded model? A single
Configuration is being used or new ones are being created per servlet
invocation?
Below is code and output.
(For better perf, cache the connection)
St.Ack
1 package org.apache.hadoop.hbase;
2
3 import java.io.IOException;
4
5 import org.apache.hadoop.conf.Configuration;
6 import org.apache.hadoop.hbase.client.HConnection;
7 import org.apache.hadoop.hbase.client.HConnectionManager;
8 import org.apache.hadoop.hbase.client.HTableInterface;
9 import org.apache.hadoop.hbase.client.Put;
10 import org.apache.hadoop.hbase.util.Bytes;
11
12 public class TestConnnection {
13 public static void main(String[] args) throws IOException {
14 Configuration config = HBaseConfiguration.create();
15 for (int i = 0; i < 1000; i++) {
16 HConnection connection =
HConnectionManager.createConnection(config);
17 HTableInterface table =
connection.getTable(TableName.valueOf("table1"));
18 byte [] cf = Bytes.toBytes("t");
19 try {
20 byte [] bytes = Bytes.toBytes(i);
21 Put p = new Put(bytes);
22 p.add(cf, cf, bytes);
23 table.put(p);
24 } finally {
25 table.close();
26 connection.close();
27 }
28 System.out.println("i=" + i);
29 }
30 }
31 }
....
2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
ensemble=localhost:2181
2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=90000
watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Opening socket connection to server localhost/
127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Socket connection established to localhost/
127.0.0.1:2181, initiating session
2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Session establishment complete on server localhost/
127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout = 40000
2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: Finished
with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
STARTKEY => '', ENDKEY => ''}
2014-12-12 11:26:10,406 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x14a3fe0664b083a
2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
0x14a3fe0664b083a closed
2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
i=997
2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
ensemble=localhost:2181
2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=90000
watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL
(unknown error)
2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Socket connection established to
localhost/0:0:0:0:0:0:0:1:2181, initiating session
2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, negotiated
timeout = 40000
2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: Finished
with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
STARTKEY => '', ENDKEY => ''}
2014-12-12 11:26:10,521 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x14a3fe0664b083b
2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
0x14a3fe0664b083b closed
2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
i=998
2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
ensemble=localhost:2181
2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=90000
watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Opening socket connection to server localhost/
127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Socket connection established to localhost/
127.0.0.1:2181, initiating session
2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Session establishment complete on server localhost/
127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout = 40000
2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: Finished
with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
STARTKEY => '', ENDKEY => ''}
2014-12-12 11:26:10,638 INFO [main]
client.HConnectionManager$HConnectionImplementation: Closing zookeeper
sessionid=0x14a3fe0664b083c
2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
0x14a3fe0664b083c closed
2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
i=999
On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, I'm using CDH 5.2, 0.98
> I don't know how to use it correctly. I've just used this sample:
>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>
> HConnection connection = HConnectionManager.createConnection(config);
> HTableInterface table = connection.getTable(TableName.valueOf("table1"));
> try {
> // Use the table as needed, for a single operation and a single thread
> } finally {
> table.close();
> connection.close();
> }
>
>
> Functional testing OK, performance fails :((((
>
>
>
> 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
> >
> > Does zk count go up on each request to the servlet? Which version of
> hbase
> > so can try on this end? Do you have some client-side log? Better if you
> > cache the connection rather than make it each time since setup is costly
> > but lets fix first problem first.
> > St.Ack
> >
> > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> serega.sheypak@gmail.com>
> > wrote:
> >
> > > Hi, I'm using HConnectionManager from java servlet
> > > Looks like it's leaking, all my zookeepers complains that there is too
> > many
> > > connections from servlet hosts.
> > > Typical line from lZK log:
> > >
> > > oo many connections from /my.tomcat.server.com - max is 60
> > >
> > >
> > > Here is code sample
> > >
> > > public class BaseConnection {
> > >
> > > private static final Logger LOG =
> > > LoggerFactory.getLogger(BaseConnection.class);
> > >
> > > protected void close(HConnection connection){
> > > try{
> > > if(connection == null){
> > > return;
> > > }
> > > connection.close();
> > > }
> > > catch (Exception e){
> > > LOG.warn("Error while closing HConnection", e);
> > > }
> > > }
> > >
> > > protected void close(HTableInterface hTable){
> > > try{
> > > if(hTable == null){
> > > return;
> > > }
> > > hTable.close();
> > > }
> > > catch (Exception e){
> > > LOG.warn("Error while closing HTable", e);
> > > }
> > > }
> > > }
> > >
> > > sample PUT code from subclass:
> > >
> > > public void put(List<MyBean> entries){
> > > HConnection hConnection = null;
> > > HTableInterface hTable = null;
> > > try {
> > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > for(MyBean myBean : entries){
> > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > }
> > > hConnection =
> HConnectionManager.createConnection(configuration);
> > > hTable = hConnection.getTable(NAME_B);
> > > hTable.put(puts);
> > >
> > > }catch (Exception e){
> > > LOG.error("Error while doing bulk put", e);
> > > }finally{
> > > close(hTable);
> > > close(hConnection);
> > > }
> > > }
> > >
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Serega Sheypak <se...@gmail.com>.
Hi, I'm using CDH 5.2, 0.98
I don't know how to use it correctly. I've just used this sample:
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface table = connection.getTable(TableName.valueOf("table1"));
try {
// Use the table as needed, for a single operation and a single thread
} finally {
table.close();
connection.close();
}
Functional testing OK, performance fails :((((
2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>:
>
> Does zk count go up on each request to the servlet? Which version of hbase
> so can try on this end? Do you have some client-side log? Better if you
> cache the connection rather than make it each time since setup is costly
> but lets fix first problem first.
> St.Ack
>
> On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, I'm using HConnectionManager from java servlet
> > Looks like it's leaking, all my zookeepers complains that there is too
> many
> > connections from servlet hosts.
> > Typical line from lZK log:
> >
> > oo many connections from /my.tomcat.server.com - max is 60
> >
> >
> > Here is code sample
> >
> > public class BaseConnection {
> >
> > private static final Logger LOG =
> > LoggerFactory.getLogger(BaseConnection.class);
> >
> > protected void close(HConnection connection){
> > try{
> > if(connection == null){
> > return;
> > }
> > connection.close();
> > }
> > catch (Exception e){
> > LOG.warn("Error while closing HConnection", e);
> > }
> > }
> >
> > protected void close(HTableInterface hTable){
> > try{
> > if(hTable == null){
> > return;
> > }
> > hTable.close();
> > }
> > catch (Exception e){
> > LOG.warn("Error while closing HTable", e);
> > }
> > }
> > }
> >
> > sample PUT code from subclass:
> >
> > public void put(List<MyBean> entries){
> > HConnection hConnection = null;
> > HTableInterface hTable = null;
> > try {
> > List<Put> puts = new ArrayList<Put>(entries.size());
> > for(MyBean myBean : entries){
> > puts.add(new MyBeanSerDe().createPut(myBean));
> > }
> > hConnection = HConnectionManager.createConnection(configuration);
> > hTable = hConnection.getTable(NAME_B);
> > hTable.put(puts);
> >
> > }catch (Exception e){
> > LOG.error("Error while doing bulk put", e);
> > }finally{
> > close(hTable);
> > close(hConnection);
> > }
> > }
> >
>
Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Posted by Stack <st...@duboce.net>.
Does zk count go up on each request to the servlet? Which version of hbase
so can try on this end? Do you have some client-side log? Better if you
cache the connection rather than make it each time since setup is costly
but lets fix first problem first.
St.Ack
On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, I'm using HConnectionManager from java servlet
> Looks like it's leaking, all my zookeepers complains that there is too many
> connections from servlet hosts.
> Typical line from lZK log:
>
> oo many connections from /my.tomcat.server.com - max is 60
>
>
> Here is code sample
>
> public class BaseConnection {
>
> private static final Logger LOG =
> LoggerFactory.getLogger(BaseConnection.class);
>
> protected void close(HConnection connection){
> try{
> if(connection == null){
> return;
> }
> connection.close();
> }
> catch (Exception e){
> LOG.warn("Error while closing HConnection", e);
> }
> }
>
> protected void close(HTableInterface hTable){
> try{
> if(hTable == null){
> return;
> }
> hTable.close();
> }
> catch (Exception e){
> LOG.warn("Error while closing HTable", e);
> }
> }
> }
>
> sample PUT code from subclass:
>
> public void put(List<MyBean> entries){
> HConnection hConnection = null;
> HTableInterface hTable = null;
> try {
> List<Put> puts = new ArrayList<Put>(entries.size());
> for(MyBean myBean : entries){
> puts.add(new MyBeanSerDe().createPut(myBean));
> }
> hConnection = HConnectionManager.createConnection(configuration);
> hTable = hConnection.getTable(NAME_B);
> hTable.put(puts);
>
> }catch (Exception e){
> LOG.error("Error while doing bulk put", e);
> }finally{
> close(hTable);
> close(hConnection);
> }
> }
>