You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Serega Sheypak <se...@gmail.com> on 2014/08/04 19:44:10 UTC
Connection pool Concurrency in HBase
Hi, I'm trying to understand how does connection pooling works in HBase.
I've seen that
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
is recommended to use.
I have a servlet, it's instance shaed among many threads.
What is a good way to use connection pooling in this case?
Is this
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface table = connection.getTable("table1");
try {
// Use the table as needed, for a single operation and a single thread
} finally {
table.close();
connection.close();
}
1. enough to reuse connection and they wouldn't be opened each time?
2. why do I have to close ALL: table and connection? It's done by design?
Re: Connection pool Concurrency in HBase
Posted by Dhaval Shah <pr...@yahoo.co.in>.
HConnection connection = HConnectionManager.createConnection(config); will give you the shared HConnection.
Do not close the connection object until all your threads are done using it. In your use case you should not close it when you close the table since other threads may be using it or may need to use it in the future
Regards,
Dhaval
________________________________
From: Serega Sheypak <se...@gmail.com>
To: user@hbase.apache.org
Sent: Monday, 4 August 2014 1:44 PM
Subject: Connection pool Concurrency in HBase
Hi, I'm trying to understand how does connection pooling works in HBase.
I've seen that
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
is recommended to use.
I have a servlet, it's instance shaed among many threads.
What is a good way to use connection pooling in this case?
Is this
HConnection connection = HConnectionManager.createConnection(config);
HTableInterface table = connection.getTable("table1");
try {
// Use the table as needed, for a single operation and a single thread
} finally {
table.close();
connection.close();
}
1. enough to reuse connection and they wouldn't be opened each time?
2. why do I have to close ALL: table and connection? It's done by design?
Re: Connection pool Concurrency in HBase
Posted by Serega Sheypak <se...@gmail.com>.
Omg, there was a problem with a system which prevents DDoS.
Everything works fine.
2014-08-18 17:07 GMT+04:00 Serega Sheypak <se...@gmail.com>:
> I do get 1..100 cells using key+family+qualifier in single request.
> then I serialize HBase result to json using my domain model and send it to
> browser.
> General idea is to reuse connections and do not repeat full connection
> path: quorum, root, meta, RS with data for each servlet request. HBase
> client with connection can cache it. That is why I think that
> HConnectionManager can help me reuse connection in concurrency
>
>
> 2014-08-18 15:53 GMT+04:00 Mingtao Zhang <ma...@gmail.com>:
>
> Hi Serega,
>>
>> What's your purpose related to connection pool in doGet?
>>
>> Mingtao Sent from iPhone
>>
>> > On Aug 18, 2014, at 4:49 AM, Serega Sheypak <se...@gmail.com>
>> wrote:
>> >
>> > I have several servlets https://en.wikipedia.org/wiki/Java_Servlet.
>> > Servlet have a lifecycle: it's " public void init(...)" method called
>> only
>> > once during container startup. Servlet has SINGLE instance shared among
>> > many web-threads.
>> > I want to init connection pool in servlet init method and then use this
>> > pool in doGet method of servlet.
>> >
>> >
>> >
>> > 2014-08-18 5:13 GMT+04:00 Mingtao Zhang <ma...@gmail.com>:
>> >
>> >> We may need to know what you are planning to do with the connection
>> pool
>> >> to help.
>> >>
>> >> Mingtao Sent from iPhone
>> >>
>> >>>> On Aug 4, 2014, at 1:44 PM, Serega Sheypak <serega.sheypak@gmail.com
>> >
>> >>> wrote:
>> >>>
>> >>> Hi, I'm trying to understand how does connection pooling works in
>> HBase.
>> >>> I've seen that
>> >>
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>> >>> is recommended to use.
>> >>>
>> >>> I have a servlet, it's instance shaed among many threads.
>> >>> What is a good way to use connection pooling in this case?
>> >>>
>> >>> Is this
>> >>>
>> >>> HConnection connection = HConnectionManager.createConnection(config);
>> >>> HTableInterface table = connection.getTable("table1");
>> >>> try {
>> >>> // Use the table as needed, for a single operation and a single
>> thread
>> >>> } finally {
>> >>> table.close();
>> >>> connection.close();
>> >>> }
>> >>>
>> >>>
>> >>> 1. enough to reuse connection and they wouldn't be opened each time?
>> >>> 2. why do I have to close ALL: table and connection? It's done by
>> design?
>> >>
>>
>
>
Re: Connection pool Concurrency in HBase
Posted by Serega Sheypak <se...@gmail.com>.
I do get 1..100 cells using key+family+qualifier in single request.
then I serialize HBase result to json using my domain model and send it to
browser.
General idea is to reuse connections and do not repeat full connection
path: quorum, root, meta, RS with data for each servlet request. HBase
client with connection can cache it. That is why I think that
HConnectionManager can help me reuse connection in concurrency
2014-08-18 15:53 GMT+04:00 Mingtao Zhang <ma...@gmail.com>:
> Hi Serega,
>
> What's your purpose related to connection pool in doGet?
>
> Mingtao Sent from iPhone
>
> > On Aug 18, 2014, at 4:49 AM, Serega Sheypak <se...@gmail.com>
> wrote:
> >
> > I have several servlets https://en.wikipedia.org/wiki/Java_Servlet.
> > Servlet have a lifecycle: it's " public void init(...)" method called
> only
> > once during container startup. Servlet has SINGLE instance shared among
> > many web-threads.
> > I want to init connection pool in servlet init method and then use this
> > pool in doGet method of servlet.
> >
> >
> >
> > 2014-08-18 5:13 GMT+04:00 Mingtao Zhang <ma...@gmail.com>:
> >
> >> We may need to know what you are planning to do with the connection pool
> >> to help.
> >>
> >> Mingtao Sent from iPhone
> >>
> >>>> On Aug 4, 2014, at 1:44 PM, Serega Sheypak <se...@gmail.com>
> >>> wrote:
> >>>
> >>> Hi, I'm trying to understand how does connection pooling works in
> HBase.
> >>> I've seen that
> >>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> >>> is recommended to use.
> >>>
> >>> I have a servlet, it's instance shaed among many threads.
> >>> What is a good way to use connection pooling in this case?
> >>>
> >>> Is this
> >>>
> >>> HConnection connection = HConnectionManager.createConnection(config);
> >>> HTableInterface table = connection.getTable("table1");
> >>> try {
> >>> // Use the table as needed, for a single operation and a single thread
> >>> } finally {
> >>> table.close();
> >>> connection.close();
> >>> }
> >>>
> >>>
> >>> 1. enough to reuse connection and they wouldn't be opened each time?
> >>> 2. why do I have to close ALL: table and connection? It's done by
> design?
> >>
>
Re: Connection pool Concurrency in HBase
Posted by Mingtao Zhang <ma...@gmail.com>.
Hi Serega,
What's your purpose related to connection pool in doGet?
Mingtao Sent from iPhone
> On Aug 18, 2014, at 4:49 AM, Serega Sheypak <se...@gmail.com> wrote:
>
> I have several servlets https://en.wikipedia.org/wiki/Java_Servlet.
> Servlet have a lifecycle: it's " public void init(...)" method called only
> once during container startup. Servlet has SINGLE instance shared among
> many web-threads.
> I want to init connection pool in servlet init method and then use this
> pool in doGet method of servlet.
>
>
>
> 2014-08-18 5:13 GMT+04:00 Mingtao Zhang <ma...@gmail.com>:
>
>> We may need to know what you are planning to do with the connection pool
>> to help.
>>
>> Mingtao Sent from iPhone
>>
>>>> On Aug 4, 2014, at 1:44 PM, Serega Sheypak <se...@gmail.com>
>>> wrote:
>>>
>>> Hi, I'm trying to understand how does connection pooling works in HBase.
>>> I've seen that
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>>> is recommended to use.
>>>
>>> I have a servlet, it's instance shaed among many threads.
>>> What is a good way to use connection pooling in this case?
>>>
>>> Is this
>>>
>>> HConnection connection = HConnectionManager.createConnection(config);
>>> HTableInterface table = connection.getTable("table1");
>>> try {
>>> // Use the table as needed, for a single operation and a single thread
>>> } finally {
>>> table.close();
>>> connection.close();
>>> }
>>>
>>>
>>> 1. enough to reuse connection and they wouldn't be opened each time?
>>> 2. why do I have to close ALL: table and connection? It's done by design?
>>
Re: Connection pool Concurrency in HBase
Posted by Serega Sheypak <se...@gmail.com>.
I have several servlets https://en.wikipedia.org/wiki/Java_Servlet.
Servlet have a lifecycle: it's " public void init(...)" method called only
once during container startup. Servlet has SINGLE instance shared among
many web-threads.
I want to init connection pool in servlet init method and then use this
pool in doGet method of servlet.
2014-08-18 5:13 GMT+04:00 Mingtao Zhang <ma...@gmail.com>:
> We may need to know what you are planning to do with the connection pool
> to help.
>
> Mingtao Sent from iPhone
>
> > On Aug 4, 2014, at 1:44 PM, Serega Sheypak <se...@gmail.com>
> wrote:
> >
> > Hi, I'm trying to understand how does connection pooling works in HBase.
> > I've seen that
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > is recommended to use.
> >
> > I have a servlet, it's instance shaed among many threads.
> > What is a good way to use connection pooling in this case?
> >
> > Is this
> >
> > HConnection connection = HConnectionManager.createConnection(config);
> > HTableInterface table = connection.getTable("table1");
> > try {
> > // Use the table as needed, for a single operation and a single thread
> > } finally {
> > table.close();
> > connection.close();
> > }
> >
> >
> > 1. enough to reuse connection and they wouldn't be opened each time?
> > 2. why do I have to close ALL: table and connection? It's done by design?
>
Re: Connection pool Concurrency in HBase
Posted by Mingtao Zhang <ma...@gmail.com>.
We may need to know what you are planning to do with the connection pool to help.
Mingtao Sent from iPhone
> On Aug 4, 2014, at 1:44 PM, Serega Sheypak <se...@gmail.com> wrote:
>
> Hi, I'm trying to understand how does connection pooling works in HBase.
> I've seen that
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> is recommended to use.
>
> I have a servlet, it's instance shaed among many threads.
> What is a good way to use connection pooling in this case?
>
> Is this
>
> HConnection connection = HConnectionManager.createConnection(config);
> HTableInterface table = connection.getTable("table1");
> try {
> // Use the table as needed, for a single operation and a single thread
> } finally {
> table.close();
> connection.close();
> }
>
>
> 1. enough to reuse connection and they wouldn't be opened each time?
> 2. why do I have to close ALL: table and connection? It's done by design?
Re: Connection pool Concurrency in HBase
Posted by Serega Sheypak <se...@gmail.com>.
Hi, let's wait for a few days. It's possible that the problem is in
infrastructure. We need time to inspect this trouble before trying to fix
HConnectionManager.
2014-08-18 2:33 GMT+04:00 Anil Gupta <an...@gmail.com>:
> Hi Serega,
>
> I am unable to figure what you are doing. Can you please share your code?
> Or explain in more details about your implementation?
>
> Sent from my iPhone
>
> > On Aug 17, 2014, at 12:36 PM, Serega Sheypak <se...@gmail.com>
> wrote:
> >
> > It doesn't work.
> > I do get in logs:
> > 23:15:27,579 [http-8080-1-SendThread(zookeeper-02.my.ru:2181)] DEBUG Got
> > ping response for sessionid: 0x246b06d6b0ae108 after 0ms
> > And tha's all I don't get exceptions and response. I use GET operation,
> no
> > scans or heavy stuff like that.
> >
> >
> >
> > 2014-08-04 22:09 GMT+04:00 Serega Sheypak <se...@gmail.com>:
> >
> >> Thanks, I'll read and inspect.
> >>
> >>
> >> 2014-08-04 21:52 GMT+04:00 anil gupta <an...@gmail.com>:
> >>
> >> Hi Serega,
> >>>
> >>> We have been using this constructor in HBase0.94:
> >>>
> >>>
> https://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html#HTable%28byte[],%20org.apache.hadoop.hbase.client.HConnection,%20java.util.concurrent.ExecutorService%29
> >>>
> >>> This constructor is pretty light weight. But, you will need to manage
> >>> ExecutorService and HConnection by yourself. We create these in our
> >>> context
> >>> at startup and keep on reusing.
> >>> We use this in our web services and we call the HTable constructor for
> >>> every request separately so you will not run into threading issue.
> >>> NOTE: I havent tested this constructor for doing Writes through
> >>> webservices. We have using this for reads.
> >>>
> >>> HTH,
> >>> Anil Gupta
> >>>
> >>>
> >>>
> >>> On Mon, Aug 4, 2014 at 10:44 AM, Serega Sheypak <
> serega.sheypak@gmail.com
> >>> wrote:
> >>>
> >>>> Hi, I'm trying to understand how does connection pooling works in
> HBase.
> >>>> I've seen that
> >>>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> >>>> is recommended to use.
> >>>>
> >>>> I have a servlet, it's instance shaed among many threads.
> >>>> What is a good way to use connection pooling in this case?
> >>>>
> >>>> Is this
> >>>>
> >>>> HConnection connection = HConnectionManager.createConnection(config);
> >>>> HTableInterface table = connection.getTable("table1");
> >>>> try {
> >>>> // Use the table as needed, for a single operation and a single
> >>> thread
> >>>> } finally {
> >>>> table.close();
> >>>> connection.close();
> >>>> }
> >>>>
> >>>>
> >>>> 1. enough to reuse connection and they wouldn't be opened each time?
> >>>> 2. why do I have to close ALL: table and connection? It's done by
> >>> design?
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Anil Gupta
> >>
> >>
>
Re: Connection pool Concurrency in HBase
Posted by Anil Gupta <an...@gmail.com>.
Hi Serega,
I am unable to figure what you are doing. Can you please share your code? Or explain in more details about your implementation?
Sent from my iPhone
> On Aug 17, 2014, at 12:36 PM, Serega Sheypak <se...@gmail.com> wrote:
>
> It doesn't work.
> I do get in logs:
> 23:15:27,579 [http-8080-1-SendThread(zookeeper-02.my.ru:2181)] DEBUG Got
> ping response for sessionid: 0x246b06d6b0ae108 after 0ms
> And tha's all I don't get exceptions and response. I use GET operation, no
> scans or heavy stuff like that.
>
>
>
> 2014-08-04 22:09 GMT+04:00 Serega Sheypak <se...@gmail.com>:
>
>> Thanks, I'll read and inspect.
>>
>>
>> 2014-08-04 21:52 GMT+04:00 anil gupta <an...@gmail.com>:
>>
>> Hi Serega,
>>>
>>> We have been using this constructor in HBase0.94:
>>>
>>> https://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html#HTable%28byte[],%20org.apache.hadoop.hbase.client.HConnection,%20java.util.concurrent.ExecutorService%29
>>>
>>> This constructor is pretty light weight. But, you will need to manage
>>> ExecutorService and HConnection by yourself. We create these in our
>>> context
>>> at startup and keep on reusing.
>>> We use this in our web services and we call the HTable constructor for
>>> every request separately so you will not run into threading issue.
>>> NOTE: I havent tested this constructor for doing Writes through
>>> webservices. We have using this for reads.
>>>
>>> HTH,
>>> Anil Gupta
>>>
>>>
>>>
>>> On Mon, Aug 4, 2014 at 10:44 AM, Serega Sheypak <serega.sheypak@gmail.com
>>> wrote:
>>>
>>>> Hi, I'm trying to understand how does connection pooling works in HBase.
>>>> I've seen that
>>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>>>> is recommended to use.
>>>>
>>>> I have a servlet, it's instance shaed among many threads.
>>>> What is a good way to use connection pooling in this case?
>>>>
>>>> Is this
>>>>
>>>> HConnection connection = HConnectionManager.createConnection(config);
>>>> HTableInterface table = connection.getTable("table1");
>>>> try {
>>>> // Use the table as needed, for a single operation and a single
>>> thread
>>>> } finally {
>>>> table.close();
>>>> connection.close();
>>>> }
>>>>
>>>>
>>>> 1. enough to reuse connection and they wouldn't be opened each time?
>>>> 2. why do I have to close ALL: table and connection? It's done by
>>> design?
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>
>>
Re: Connection pool Concurrency in HBase
Posted by Serega Sheypak <se...@gmail.com>.
It doesn't work.
I do get in logs:
23:15:27,579 [http-8080-1-SendThread(zookeeper-02.my.ru:2181)] DEBUG Got
ping response for sessionid: 0x246b06d6b0ae108 after 0ms
And tha's all I don't get exceptions and response. I use GET operation, no
scans or heavy stuff like that.
2014-08-04 22:09 GMT+04:00 Serega Sheypak <se...@gmail.com>:
> Thanks, I'll read and inspect.
>
>
> 2014-08-04 21:52 GMT+04:00 anil gupta <an...@gmail.com>:
>
> Hi Serega,
>>
>> We have been using this constructor in HBase0.94:
>>
>> https://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html#HTable%28byte[],%20org.apache.hadoop.hbase.client.HConnection,%20java.util.concurrent.ExecutorService%29
>>
>> This constructor is pretty light weight. But, you will need to manage
>> ExecutorService and HConnection by yourself. We create these in our
>> context
>> at startup and keep on reusing.
>> We use this in our web services and we call the HTable constructor for
>> every request separately so you will not run into threading issue.
>> NOTE: I havent tested this constructor for doing Writes through
>> webservices. We have using this for reads.
>>
>> HTH,
>> Anil Gupta
>>
>>
>>
>> On Mon, Aug 4, 2014 at 10:44 AM, Serega Sheypak <serega.sheypak@gmail.com
>> >
>> wrote:
>>
>> > Hi, I'm trying to understand how does connection pooling works in HBase.
>> > I've seen that
>> >
>> >
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>> > is recommended to use.
>> >
>> > I have a servlet, it's instance shaed among many threads.
>> > What is a good way to use connection pooling in this case?
>> >
>> > Is this
>> >
>> > HConnection connection = HConnectionManager.createConnection(config);
>> > HTableInterface table = connection.getTable("table1");
>> > try {
>> > // Use the table as needed, for a single operation and a single
>> thread
>> > } finally {
>> > table.close();
>> > connection.close();
>> > }
>> >
>> >
>> > 1. enough to reuse connection and they wouldn't be opened each time?
>> > 2. why do I have to close ALL: table and connection? It's done by
>> design?
>> >
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>
>
Re: Connection pool Concurrency in HBase
Posted by Serega Sheypak <se...@gmail.com>.
Thanks, I'll read and inspect.
2014-08-04 21:52 GMT+04:00 anil gupta <an...@gmail.com>:
> Hi Serega,
>
> We have been using this constructor in HBase0.94:
>
> https://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html#HTable%28byte[],%20org.apache.hadoop.hbase.client.HConnection,%20java.util.concurrent.ExecutorService%29
>
> This constructor is pretty light weight. But, you will need to manage
> ExecutorService and HConnection by yourself. We create these in our context
> at startup and keep on reusing.
> We use this in our web services and we call the HTable constructor for
> every request separately so you will not run into threading issue.
> NOTE: I havent tested this constructor for doing Writes through
> webservices. We have using this for reads.
>
> HTH,
> Anil Gupta
>
>
>
> On Mon, Aug 4, 2014 at 10:44 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, I'm trying to understand how does connection pooling works in HBase.
> > I've seen that
> >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > is recommended to use.
> >
> > I have a servlet, it's instance shaed among many threads.
> > What is a good way to use connection pooling in this case?
> >
> > Is this
> >
> > HConnection connection = HConnectionManager.createConnection(config);
> > HTableInterface table = connection.getTable("table1");
> > try {
> > // Use the table as needed, for a single operation and a single thread
> > } finally {
> > table.close();
> > connection.close();
> > }
> >
> >
> > 1. enough to reuse connection and they wouldn't be opened each time?
> > 2. why do I have to close ALL: table and connection? It's done by design?
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Connection pool Concurrency in HBase
Posted by anil gupta <an...@gmail.com>.
Hi Serega,
We have been using this constructor in HBase0.94:
https://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HTable.html#HTable%28byte[],%20org.apache.hadoop.hbase.client.HConnection,%20java.util.concurrent.ExecutorService%29
This constructor is pretty light weight. But, you will need to manage
ExecutorService and HConnection by yourself. We create these in our context
at startup and keep on reusing.
We use this in our web services and we call the HTable constructor for
every request separately so you will not run into threading issue.
NOTE: I havent tested this constructor for doing Writes through
webservices. We have using this for reads.
HTH,
Anil Gupta
On Mon, Aug 4, 2014 at 10:44 AM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, I'm trying to understand how does connection pooling works in HBase.
> I've seen that
>
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> is recommended to use.
>
> I have a servlet, it's instance shaed among many threads.
> What is a good way to use connection pooling in this case?
>
> Is this
>
> HConnection connection = HConnectionManager.createConnection(config);
> HTableInterface table = connection.getTable("table1");
> try {
> // Use the table as needed, for a single operation and a single thread
> } finally {
> table.close();
> connection.close();
> }
>
>
> 1. enough to reuse connection and they wouldn't be opened each time?
> 2. why do I have to close ALL: table and connection? It's done by design?
>
--
Thanks & Regards,
Anil Gupta