You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by lars hofhansl <la...@apache.org> on 2013/08/03 06:29:50 UTC
Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
See. https://issues.apache.org/jira/browse/HBASE-6580
The new proposed API looks like this:
Here's the proposed new API:
* HConnectionManager:
public static HConnection createConnection(Configuration conf)
public static HConnection createConnection(Configuration conf, ExecutorService pool)
* HConnection:
public HTableInterface getTable(byte[] tableName) throws IOException
public HTableInterface getTable(byte[] tableName, ExecutorService pool) throws IOException
public HTableInterface getTable(String tableName) throws IOException
By default HConnectionImplementation will create an ExecutorService when needed. The ExecutorService can optionally passed be passed in.
HTableInterfaces are retrieved from the HConnection. By default the HConnection's ExecutorService is used, but optionally that can be overridden for each HTable.
In 0.98/trunk:
1. HTablePool will be removed. It is not longer needed.
2. All constructors in HTable will be removed and changed to be protected. All code use HTableInterface only.
3. HConnectionManager.getConnection() will be removed.
3. All HConnection caching (deleteConnection, etc,etc) will be removed, as it is no longer needed.
The new flow of setting up a client would look like this:
----- Snip -----
// connection to the cluster
HConnection conn = HConnectionManager.createConnection(conf);
...
// When the cluster connection is established get an HTableInterface for each operation or thread.
// HConnection.getTable(...) is lightweight. The table is really just a convenient place to call table method and for a temporary batch cache.
// It is in fact less overhead than HTablePool had when retrieving a cached HTable.
// The HTableInterface returned is not thread safe as before.
// It's fine to get 1000's of these.
// Don't cache the longer than the lifetime of the HConnection
HTableInterface table = conn.getTable("MyTable");
...
// just flushes outstanding commit, no futher cleanup needed, can be omitted.
// HConnection holds no references to the returned HTable objects, they can be GC'd as soon as they leave scope.
table.close();
...
conn.close(); // done with the cluster, release resources
----- Snip -----
The HConnection will maintain and share its own ThreadPool for all batch operations executed by the HTables.
This can overridden per HConnection and/or per individual HTable object.
I will commit the new API to all branches early next week.
Questions? Comments? Concerns? Praise?
-- Lars
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Nick Dimiduk <nd...@gmail.com>.
On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
> * HConnection:
> public HTableInterface getTable(byte[] tableName) throws IOException
> public HTableInterface getTable(byte[] tableName, ExecutorService
> pool) throws IOException
> public HTableInterface getTable(String tableName) throws IOException
>
Mind adding `public HTableInterface getTable(String tableName,
ExecutorService pool) throws IOException` to the permutations?
HTableInterfaces are retrieved from the HConnection. By default the
> HConnection's ExecutorService is used, but optionally that can be
> overridden for each HTable.
>
This sounds great!
// HConnection.getTable(...) is lightweight. The table is really just a
> convenient place to call table method and for a temporary batch cache.
>
Do the semantics of the batch cache change at all, with respect to
accumulating mutations per Table/RegionServer/Region?
// The HTableInterface returned is not thread safe as before.
>
I haven't looked at the patch -- can this be placed in big bold letters in
the class and/or constructor javadoc?
// It's fine to get 1000's of these.
>
Nice!
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Jesse Yates <je...@gmail.com>.
This kind of code snippet would be a great addition to the ref guide too -
people are definitely going to ask (apologies if you already did it in the
patch, I didn't look).
- jesse
/typed on a silly-small keyboard
On Friday, August 2, 2013, lars hofhansl wrote:
> Yeah, I filed a separate ticket for the API removal in trunk.
>
>
>
> ________________________________
> From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> To: dev@hbase.apache.org <javascript:;>; lars hofhansl <larsh@apache.org<javascript:;>
> >
> Sent: Friday, August 2, 2013 10:31 PM
> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
>
>
> bq. HConnectionManager.getConnection() will be removed.
>
> I don't see the above change in 6580-trunk.txt
> Would the above be done in next patch or in another JIRA ?
>
> Cheers
>
> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
>
> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >
> > The new proposed API looks like this:
> >
> > Here's the proposed new API:
> > * HConnectionManager:
> > public static HConnection createConnection(Configuration conf)
> > public static HConnection createConnection(Configuration conf,
> > ExecutorService pool)
> >
> > * HConnection:
> > public HTableInterface getTable(byte[] tableName) throws IOException
> > public HTableInterface getTable(byte[] tableName, ExecutorService
> > pool) throws IOException
> > public HTableInterface getTable(String tableName) throws IOException
> >
> > By default HConnectionImplementation will create an ExecutorService when
> > needed. The ExecutorService can optionally passed be passed in.
> > HTableInterfaces are retrieved from the HConnection. By default the
> > HConnection's ExecutorService is used, but optionally that can be
> > overridden for each HTable.
> >
> > In 0.98/trunk:
> >
> > 1. HTablePool will be removed. It is not longer needed.
> > 2. All constructors in HTable will be removed and changed to be
> protected.
> > All code use HTableInterface only.
> > 3. HConnectionManager.getConnection() will be removed.
> > 3. All HConnection caching (deleteConnection, etc,etc) will be removed,
> as
> > it is no longer needed.
> >
> >
> > The new flow of setting up a client would look like this:
> >
> > ----- Snip -----
> > // connection to the cluster
> > HConnection conn = HConnectionManager.createConnection(conf);
> > ...
> > // When the cluster connection is established get an HTableInterface for
> > each operation or thread.
> > // HConnection.getTable(...) is lightweight. The table is really just a
> > convenient place to call table method and for a temporary batch cache.
> > // It is in fact less overhead than HTablePool had when retrieving a
> > cached HTable.
> > // The HTableInterface returned is not thread safe as before.
> > // It's fine to get 1000's of these.
> > // Don't cache the longer than the lifetime of the HConnection
> > HTableInterface table = conn.getTable("MyTable");
> > ...
> > // just flushes outstanding commit, no futher cleanup needed, can be
> > omitted.
> > // HConnection holds no references to the returned HTable objects, they
> > can be GC'd as soon as they leave scope.
> > table.close();
> > ...
> > conn.close(); // done with the cluster, release resources
> > ----- Snip -----
> >
> > The HConnection will maintain and share its own ThreadPool for all batch
> > operations executed by the HTables.
> > This can overridden per HConnection and/or per individual HTable object.
> >
> > I will commit the new API to all branches early next week.
> >
> > Questions? Comments? Concerns? Praise?
> >
> > -- Lars
--
-------------------
Jesse Yates
@jesse_yates
jyates.github.com
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
+1 as well.
On Mon, Aug 5, 2013 at 2:30 PM, Enis Söztutar <en...@gmail.com> wrote:
> +1 on making the connection API explicit.
>
> We should start dog-fooding, and actively encouraging the new usage in 0.96
> (can come after 0.96.0)
>
> Enis
>
>
> On Mon, Aug 5, 2013 at 1:21 PM, Suraj Varma <sv...@gmail.com> wrote:
>
> > Might this be a good time to _not_ throw IOException and perhaps throw
> > something along the lines of retryable / non-retryable etc exceptions -
> > similar to the hierarchy in asynchbase?
> >
> > Since clients have to change anyways ... perhaps it is a good time to
> > introduce this change?
> > --S
> >
> >
> > On Mon, Aug 5, 2013 at 10:45 AM, Gary Helmling <gh...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > Nice work on this Lars!
> > >
> > > This will make the client connection code a lot simpler and a lot
> easier
> > to
> > > reason about.
> > >
> > > While it's unfortunate that external client code will necessarily need
> to
> > > be reworked for the changes, I think the result will be much cleaner
> all
> > > around. It will be great to get rid of the convolutions of HTablePool
> as
> > > well. If necessary to ease the client transition, HTablePool could
> even
> > be
> > > kept, but reworked as just a simple wrapper around HConnection (no need
> > to
> > > even do reference counting, etc).
> > >
> > > Looking forward to start making use of this.
> > >
> > > --gh
> > >
> > >
> > > On Mon, Aug 5, 2013 at 10:31 AM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > +1 Lars
> > > >
> > > > I think it makes sense to take your experience with using the client
> in
> > > app
> > > > servers into API improvements.
> > > >
> > > > > Love the quiz.
> > > >
> > > > +1 nice illustration
> > > >
> > > > On Mon, Aug 5, 2013 at 8:25 AM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org>
> > > wrote:
> > > > >
> > > > > > Let's do a little quiz:
> > > > > >
> > > > > > HTable t1 = new HTable(conf);
> > > > > > t1.close();
> > > > > >
> > > > > > // 1. Will the next line create a new HConnection behind the
> scenes
> > > > > (along
> > > > > > with re-creating all the caches)?
> > > > > > // (If so, it will be expensive, if not, when is the first
> > > HConnection
> > > > > > actually released?)
> > > > > > HTable t2 = new HTable(conf);
> > > > > >
> > > > > > // 2. how about this one?
> > > > > > HTable t2 = new HTable(new Configuration(conf));
> > > > > >
> > > > > > // 3. or now?
> > > > > > conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> > > > > > HTable t3 = new HTable(conf);
> > > > > >
> > > > > > // 4. and now?
> > > > > > conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> > > > > > HTable t4 = new HTable(conf);
> > > > > >
> > > > > > // 5. how many connections are opened now?
> > > > > > t4.close();
> > > > > >
> > > > > > This stuff is convoluted and needlessly complicated. And this is
> > not
> > > > > > because the code is bad, but because the abstraction is simply
> > > > > inadequate.
> > > > > > A client wants to connect to a cluster and then do some action on
> > > that
> > > > > > cluster (via HTable as a convenience).
> > > > > > If the cluster connection is implicit it leads to all of the
> above
> > > > > > considerations.
> > > > > >
> > > > > >
> > > > > Love the quiz.
> > > > >
> > > > > +1 on your redo of our connection model (HConnection is a "cluster
> > > > > connection". I like that you have to get one of these first...)
> > > > >
> > > > > St.Ack
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
Posted by lars hofhansl <la...@apache.org>.
Thanks Andy,
as part of the patch I'll convert some of the unitests.
There are some conveniences we will no longer have such as a method that can create a table and return and HTable to it
(the connection would have to be explicit now)
-- Lars
----- Original Message -----
From: Andrew Purtell <ap...@apache.org>
To: dev@hbase.apache.org
Cc:
Sent: Monday, August 5, 2013 5:27 PM
Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
On Mon, Aug 5, 2013 at 2:30 PM, Enis Söztutar <en...@gmail.com> wrote:
> We should start dog-fooding
A small start there could be HBASE-7463
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Andrew Purtell <ap...@apache.org>.
On Mon, Aug 5, 2013 at 2:30 PM, Enis Söztutar <en...@gmail.com> wrote:
> We should start dog-fooding
A small start there could be HBASE-7463
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Enis Söztutar <en...@gmail.com>.
+1 on making the connection API explicit.
We should start dog-fooding, and actively encouraging the new usage in 0.96
(can come after 0.96.0)
Enis
On Mon, Aug 5, 2013 at 1:21 PM, Suraj Varma <sv...@gmail.com> wrote:
> Might this be a good time to _not_ throw IOException and perhaps throw
> something along the lines of retryable / non-retryable etc exceptions -
> similar to the hierarchy in asynchbase?
>
> Since clients have to change anyways ... perhaps it is a good time to
> introduce this change?
> --S
>
>
> On Mon, Aug 5, 2013 at 10:45 AM, Gary Helmling <gh...@gmail.com>
> wrote:
>
> > +1
> >
> > Nice work on this Lars!
> >
> > This will make the client connection code a lot simpler and a lot easier
> to
> > reason about.
> >
> > While it's unfortunate that external client code will necessarily need to
> > be reworked for the changes, I think the result will be much cleaner all
> > around. It will be great to get rid of the convolutions of HTablePool as
> > well. If necessary to ease the client transition, HTablePool could even
> be
> > kept, but reworked as just a simple wrapper around HConnection (no need
> to
> > even do reference counting, etc).
> >
> > Looking forward to start making use of this.
> >
> > --gh
> >
> >
> > On Mon, Aug 5, 2013 at 10:31 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > +1 Lars
> > >
> > > I think it makes sense to take your experience with using the client in
> > app
> > > servers into API improvements.
> > >
> > > > Love the quiz.
> > >
> > > +1 nice illustration
> > >
> > > On Mon, Aug 5, 2013 at 8:25 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > > >
> > > > > Let's do a little quiz:
> > > > >
> > > > > HTable t1 = new HTable(conf);
> > > > > t1.close();
> > > > >
> > > > > // 1. Will the next line create a new HConnection behind the scenes
> > > > (along
> > > > > with re-creating all the caches)?
> > > > > // (If so, it will be expensive, if not, when is the first
> > HConnection
> > > > > actually released?)
> > > > > HTable t2 = new HTable(conf);
> > > > >
> > > > > // 2. how about this one?
> > > > > HTable t2 = new HTable(new Configuration(conf));
> > > > >
> > > > > // 3. or now?
> > > > > conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> > > > > HTable t3 = new HTable(conf);
> > > > >
> > > > > // 4. and now?
> > > > > conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> > > > > HTable t4 = new HTable(conf);
> > > > >
> > > > > // 5. how many connections are opened now?
> > > > > t4.close();
> > > > >
> > > > > This stuff is convoluted and needlessly complicated. And this is
> not
> > > > > because the code is bad, but because the abstraction is simply
> > > > inadequate.
> > > > > A client wants to connect to a cluster and then do some action on
> > that
> > > > > cluster (via HTable as a convenience).
> > > > > If the cluster connection is implicit it leads to all of the above
> > > > > considerations.
> > > > >
> > > > >
> > > > Love the quiz.
> > > >
> > > > +1 on your redo of our connection model (HConnection is a "cluster
> > > > connection". I like that you have to get one of these first...)
> > > >
> > > > St.Ack
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Suraj Varma <sv...@gmail.com>.
Might this be a good time to _not_ throw IOException and perhaps throw
something along the lines of retryable / non-retryable etc exceptions -
similar to the hierarchy in asynchbase?
Since clients have to change anyways ... perhaps it is a good time to
introduce this change?
--S
On Mon, Aug 5, 2013 at 10:45 AM, Gary Helmling <gh...@gmail.com> wrote:
> +1
>
> Nice work on this Lars!
>
> This will make the client connection code a lot simpler and a lot easier to
> reason about.
>
> While it's unfortunate that external client code will necessarily need to
> be reworked for the changes, I think the result will be much cleaner all
> around. It will be great to get rid of the convolutions of HTablePool as
> well. If necessary to ease the client transition, HTablePool could even be
> kept, but reworked as just a simple wrapper around HConnection (no need to
> even do reference counting, etc).
>
> Looking forward to start making use of this.
>
> --gh
>
>
> On Mon, Aug 5, 2013 at 10:31 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > +1 Lars
> >
> > I think it makes sense to take your experience with using the client in
> app
> > servers into API improvements.
> >
> > > Love the quiz.
> >
> > +1 nice illustration
> >
> > On Mon, Aug 5, 2013 at 8:25 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org>
> wrote:
> > >
> > > > Let's do a little quiz:
> > > >
> > > > HTable t1 = new HTable(conf);
> > > > t1.close();
> > > >
> > > > // 1. Will the next line create a new HConnection behind the scenes
> > > (along
> > > > with re-creating all the caches)?
> > > > // (If so, it will be expensive, if not, when is the first
> HConnection
> > > > actually released?)
> > > > HTable t2 = new HTable(conf);
> > > >
> > > > // 2. how about this one?
> > > > HTable t2 = new HTable(new Configuration(conf));
> > > >
> > > > // 3. or now?
> > > > conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> > > > HTable t3 = new HTable(conf);
> > > >
> > > > // 4. and now?
> > > > conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> > > > HTable t4 = new HTable(conf);
> > > >
> > > > // 5. how many connections are opened now?
> > > > t4.close();
> > > >
> > > > This stuff is convoluted and needlessly complicated. And this is not
> > > > because the code is bad, but because the abstraction is simply
> > > inadequate.
> > > > A client wants to connect to a cluster and then do some action on
> that
> > > > cluster (via HTable as a convenience).
> > > > If the cluster connection is implicit it leads to all of the above
> > > > considerations.
> > > >
> > > >
> > > Love the quiz.
> > >
> > > +1 on your redo of our connection model (HConnection is a "cluster
> > > connection". I like that you have to get one of these first...)
> > >
> > > St.Ack
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Gary Helmling <gh...@gmail.com>.
+1
Nice work on this Lars!
This will make the client connection code a lot simpler and a lot easier to
reason about.
While it's unfortunate that external client code will necessarily need to
be reworked for the changes, I think the result will be much cleaner all
around. It will be great to get rid of the convolutions of HTablePool as
well. If necessary to ease the client transition, HTablePool could even be
kept, but reworked as just a simple wrapper around HConnection (no need to
even do reference counting, etc).
Looking forward to start making use of this.
--gh
On Mon, Aug 5, 2013 at 10:31 AM, Andrew Purtell <ap...@apache.org> wrote:
> +1 Lars
>
> I think it makes sense to take your experience with using the client in app
> servers into API improvements.
>
> > Love the quiz.
>
> +1 nice illustration
>
> On Mon, Aug 5, 2013 at 8:25 AM, Stack <st...@duboce.net> wrote:
>
> > On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org> wrote:
> >
> > > Let's do a little quiz:
> > >
> > > HTable t1 = new HTable(conf);
> > > t1.close();
> > >
> > > // 1. Will the next line create a new HConnection behind the scenes
> > (along
> > > with re-creating all the caches)?
> > > // (If so, it will be expensive, if not, when is the first HConnection
> > > actually released?)
> > > HTable t2 = new HTable(conf);
> > >
> > > // 2. how about this one?
> > > HTable t2 = new HTable(new Configuration(conf));
> > >
> > > // 3. or now?
> > > conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> > > HTable t3 = new HTable(conf);
> > >
> > > // 4. and now?
> > > conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> > > HTable t4 = new HTable(conf);
> > >
> > > // 5. how many connections are opened now?
> > > t4.close();
> > >
> > > This stuff is convoluted and needlessly complicated. And this is not
> > > because the code is bad, but because the abstraction is simply
> > inadequate.
> > > A client wants to connect to a cluster and then do some action on that
> > > cluster (via HTable as a convenience).
> > > If the cluster connection is implicit it leads to all of the above
> > > considerations.
> > >
> > >
> > Love the quiz.
> >
> > +1 on your redo of our connection model (HConnection is a "cluster
> > connection". I like that you have to get one of these first...)
> >
> > St.Ack
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Andrew Purtell <ap...@apache.org>.
+1 Lars
I think it makes sense to take your experience with using the client in app
servers into API improvements.
> Love the quiz.
+1 nice illustration
On Mon, Aug 5, 2013 at 8:25 AM, Stack <st...@duboce.net> wrote:
> On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Let's do a little quiz:
> >
> > HTable t1 = new HTable(conf);
> > t1.close();
> >
> > // 1. Will the next line create a new HConnection behind the scenes
> (along
> > with re-creating all the caches)?
> > // (If so, it will be expensive, if not, when is the first HConnection
> > actually released?)
> > HTable t2 = new HTable(conf);
> >
> > // 2. how about this one?
> > HTable t2 = new HTable(new Configuration(conf));
> >
> > // 3. or now?
> > conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> > HTable t3 = new HTable(conf);
> >
> > // 4. and now?
> > conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> > HTable t4 = new HTable(conf);
> >
> > // 5. how many connections are opened now?
> > t4.close();
> >
> > This stuff is convoluted and needlessly complicated. And this is not
> > because the code is bad, but because the abstraction is simply
> inadequate.
> > A client wants to connect to a cluster and then do some action on that
> > cluster (via HTable as a convenience).
> > If the cluster connection is implicit it leads to all of the above
> > considerations.
> >
> >
> Love the quiz.
>
> +1 on your redo of our connection model (HConnection is a "cluster
> connection". I like that you have to get one of these first...)
>
> St.Ack
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Stack <st...@duboce.net>.
On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org> wrote:
> Let's do a little quiz:
>
> HTable t1 = new HTable(conf);
> t1.close();
>
> // 1. Will the next line create a new HConnection behind the scenes (along
> with re-creating all the caches)?
> // (If so, it will be expensive, if not, when is the first HConnection
> actually released?)
> HTable t2 = new HTable(conf);
>
> // 2. how about this one?
> HTable t2 = new HTable(new Configuration(conf));
>
> // 3. or now?
> conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> HTable t3 = new HTable(conf);
>
> // 4. and now?
> conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> HTable t4 = new HTable(conf);
>
> // 5. how many connections are opened now?
> t4.close();
>
> This stuff is convoluted and needlessly complicated. And this is not
> because the code is bad, but because the abstraction is simply inadequate.
> A client wants to connect to a cluster and then do some action on that
> cluster (via HTable as a convenience).
> If the cluster connection is implicit it leads to all of the above
> considerations.
>
>
Love the quiz.
+1 on your redo of our connection model (HConnection is a "cluster
connection". I like that you have to get one of these first...)
St.Ack
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
Posted by lars hofhansl <la...@apache.org>.
Correct. No more guessing about what might happen behind the scenes.
________________________________
From: Ted Yu <yu...@gmail.com>
To: lars hofhansl <la...@apache.org>
Cc: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Sunday, August 4, 2013 8:08 PM
Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
w.r.t. HConstants.HBASE_CLIENT_PAUSE in #3, if user wants a different
pause, a new connection would be created explicitly in the new model, right
?
Cheers
On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org> wrote:
> Let's do a little quiz:
>
> HTable t1 = new HTable(conf);
> t1.close();
>
> // 1. Will the next line create a new HConnection behind the scenes (along
> with re-creating all the caches)?
> // (If so, it will be expensive, if not, when is the first HConnection
> actually released?)
> HTable t2 = new HTable(conf);
>
> // 2. how about this one?
> HTable t2 = new HTable(new Configuration(conf));
>
> // 3. or now?
> conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> HTable t3 = new HTable(conf);
>
> // 4. and now?
> conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> HTable t4 = new HTable(conf);
>
> // 5. how many connections are opened now?
> t4.close();
>
> This stuff is convoluted and needlessly complicated. And this is not
> because the code is bad, but because the abstraction is simply inadequate.
> A client wants to connect to a cluster and then do some action on that
> cluster (via HTable as a convenience).
> If the cluster connection is implicit it leads to all of the above
> considerations.
>
> (#1: Yes, #2: no, #3: yes, #4: no, #5: I don't really know, id'd have run
> it to see)
>
> -- Lars
>
>
>
> ------------------------------
> *From:* Ted Yu <yu...@gmail.com>
> *To:* lars hofhansl <la...@apache.org>
> *Cc:* "dev@hbase.apache.org" <de...@hbase.apache.org>
> *Sent:* Sunday, August 4, 2013 7:39 PM
>
> *Subject:* Re: Heads up, HTablePool will be deprecated in 0.94,
> 0.95/0.96, and removed in 0.98
>
> In the Connections "managing" HTables case, don't we need to figure out
> when an HConnection should be released ?
>
> On Sun, Aug 4, 2013 at 7:23 PM, lars hofhansl <la...@apache.org> wrote:
>
> Just look at HConnectionKey part, and hoops we go through to detect
> whether HConnections are the same or not, when to cache them, when/how to
> release them.
> In fact almost all HConnectionManager does is managing HConnections on
> behalf of HTable, when it should be other way around.
>
> Typically, when things get hard to explain (check out the comments in
> HConnectionManager) there is either an abstraction missing, or the
> abstraction is not right.
> The reverse (Connections "managing" HTables) has none of this.
>
>
> -- Lars
>
> _______________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Sunday, August 4, 2013 4:27 PM
> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
>
>
>
> bq. no funny business with unique Configurations
>
> Mind telling us what is funny about this part ?
>
>
> On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote:
>
> Correct. The HConnection is naturally shared between the HTables.
> >There is no longer any need to worry about this (no funny business with
> unique Configurations, in fact most of the code in HConnectionManager can
> be removed in trunk).
> >
> >It is also correct that the code now has to hold on the created
> HConnection, rather asking HConnectionManager for it.
> >
> >-- Lars
> >
> >
> >
> >________________________________
> > From: Nick Dimiduk <nd...@gmail.com>
> >To: dev@hbase.apache.org
> >Sent: Saturday, August 3, 2013 8:56 PM
> >
> >Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
> >
> >
> >On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Does this mean that user code wouldn't be able to depend
> >> on HConnectionManager for connection sharing ?
> >>
> >
> >My read of the above is that the HConnection instance is shared across
> >consumers, is the shared connection. Am I reading that correctly?
> >
> >On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Ah, I find the JIRA - HBASE-9117.
> >> >
> >> > Cheers
> >> >
> >> >
> >> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org>
> wrote:
> >> >
> >> >> Yeah, I filed a separate ticket for the API removal in trunk.
> >> >>
> >> >>
> >> >>
> >> >> ________________________________
> >> >> From: Ted Yu <yu...@gmail.com>
> >> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> >> >> Sent: Friday, August 2, 2013 10:31 PM
> >> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94,
> 0.95/0.96,
> >> >> and removed in 0.98
> >> >>
> >> >>
> >> >> bq. HConnectionManager.getConnection() will be removed.
> >> >>
> >> >> I don't see the above change in 6580-trunk.txt
> >> >> Would the above be done in next patch or in another JIRA ?
> >> >>
> >> >> Cheers
> >> >>
> >> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org>
> wrote:
> >> >>
> >> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >> >> >
> >> >> > The new proposed API looks like this:
> >> >> >
> >> >> > Here's the proposed new API:
> >> >> > * HConnectionManager:
> >> >> > public static HConnection createConnection(Configuration conf)
> >> >> > public static HConnection createConnection(Configuration conf,
> >> >> > ExecutorService pool)
> >> >> >
> >> >> > * HConnection:
> >> >> > public HTableInterface getTable(byte[] tableName) throws
> >> IOException
> >> >> > public HTableInterface getTable(byte[] tableName,
> ExecutorService
> >> >> > pool) throws IOException
> >> >> > public HTableInterface getTable(String tableName) throws
> >> IOException
> >> >> >
> >> >> > By default HConnectionImplementation will create an ExecutorService
> >> when
> >> >> > needed. The ExecutorService can optionally passed be passed in.
> >> >> > HTableInterfaces are retrieved from the HConnection. By default the
> >> >> > HConnection's ExecutorService is used, but optionally that can be
> >> >> > overridden for each HTable.
> >> >> >
> >> >> > In 0.98/trunk:
> >> >> >
> >> >> > 1. HTablePool will be removed. It is not longer needed.
> >> >> > 2. All constructors in HTable will be removed and changed to be
> >> >> protected.
> >> >> > All code use HTableInterface only.
> >> >> > 3. HConnectionManager.getConnection() will be removed.
> >> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
> >> removed,
> >> >> as
> >> >> > it is no longer needed.
> >> >> >
> >> >> >
> >> >> > The new flow of setting up a client would look like this:
> >> >> >
> >> >> > ----- Snip -----
> >> >> > // connection to the cluster
> >> >> > HConnection conn = HConnectionManager.createConnection(conf);
> >> >> > ...
> >> >> > // When the cluster connection is established get an
> HTableInterface
> >> for
> >> >> > each operation or thread.
> >> >> > // HConnection.getTable(...) is lightweight. The table is really
> just
> >> a
> >> >> > convenient place to call table method and for a temporary batch
> cache.
> >> >> > // It is in fact less overhead than HTablePool had when retrieving
> a
> >> >> > cached HTable.
> >> >> > // The HTableInterface returned is not thread safe as before.
> >> >> > // It's fine to get 1000's of these.
> >> >> > // Don't cache the longer than the lifetime of the HConnection
> >> >> > HTableInterface table = conn.getTable("MyTable");
> >> >> > ...
> >> >> > // just flushes outstanding commit, no futher cleanup needed, can
> be
> >> >> > omitted.
> >> >> > // HConnection holds no references to the returned HTable objects,
> >> they
> >> >> > can be GC'd as soon as they leave scope.
> >> >> > table.close();
> >> >> > ...
> >> >> > conn.close(); // done with the cluster, release resources
> >> >> > ----- Snip -----
> >> >> >
> >> >> > The HConnection will maintain and share its own ThreadPool for all
> >> batch
> >> >> > operations executed by the HTables.
> >> >> > This can overridden per HConnection and/or per individual HTable
> >> object.
> >> >> >
> >> >> > I will commit the new API to all branches early next week.
> >> >> >
> >> >> > Questions? Comments? Concerns? Praise?
> >> >> >
> >> >> > -- Lars
> >> >>
> >> >
> >> >
> >>
>
>
>
>
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
w.r.t. HConstants.HBASE_CLIENT_PAUSE in #3, if user wants a different
pause, a new connection would be created explicitly in the new model, right
?
Cheers
On Sun, Aug 4, 2013 at 7:56 PM, lars hofhansl <la...@apache.org> wrote:
> Let's do a little quiz:
>
> HTable t1 = new HTable(conf);
> t1.close();
>
> // 1. Will the next line create a new HConnection behind the scenes (along
> with re-creating all the caches)?
> // (If so, it will be expensive, if not, when is the first HConnection
> actually released?)
> HTable t2 = new HTable(conf);
>
> // 2. how about this one?
> HTable t2 = new HTable(new Configuration(conf));
>
> // 3. or now?
> conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
> HTable t3 = new HTable(conf);
>
> // 4. and now?
> conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
> HTable t4 = new HTable(conf);
>
> // 5. how many connections are opened now?
> t4.close();
>
> This stuff is convoluted and needlessly complicated. And this is not
> because the code is bad, but because the abstraction is simply inadequate.
> A client wants to connect to a cluster and then do some action on that
> cluster (via HTable as a convenience).
> If the cluster connection is implicit it leads to all of the above
> considerations.
>
> (#1: Yes, #2: no, #3: yes, #4: no, #5: I don't really know, id'd have run
> it to see)
>
> -- Lars
>
>
>
> ------------------------------
> *From:* Ted Yu <yu...@gmail.com>
> *To:* lars hofhansl <la...@apache.org>
> *Cc:* "dev@hbase.apache.org" <de...@hbase.apache.org>
> *Sent:* Sunday, August 4, 2013 7:39 PM
>
> *Subject:* Re: Heads up, HTablePool will be deprecated in 0.94,
> 0.95/0.96, and removed in 0.98
>
> In the Connections "managing" HTables case, don't we need to figure out
> when an HConnection should be released ?
>
> On Sun, Aug 4, 2013 at 7:23 PM, lars hofhansl <la...@apache.org> wrote:
>
> Just look at HConnectionKey part, and hoops we go through to detect
> whether HConnections are the same or not, when to cache them, when/how to
> release them.
> In fact almost all HConnectionManager does is managing HConnections on
> behalf of HTable, when it should be other way around.
>
> Typically, when things get hard to explain (check out the comments in
> HConnectionManager) there is either an abstraction missing, or the
> abstraction is not right.
> The reverse (Connections "managing" HTables) has none of this.
>
>
> -- Lars
>
> _______________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Sunday, August 4, 2013 4:27 PM
> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
>
>
>
> bq. no funny business with unique Configurations
>
> Mind telling us what is funny about this part ?
>
>
> On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote:
>
> Correct. The HConnection is naturally shared between the HTables.
> >There is no longer any need to worry about this (no funny business with
> unique Configurations, in fact most of the code in HConnectionManager can
> be removed in trunk).
> >
> >It is also correct that the code now has to hold on the created
> HConnection, rather asking HConnectionManager for it.
> >
> >-- Lars
> >
> >
> >
> >________________________________
> > From: Nick Dimiduk <nd...@gmail.com>
> >To: dev@hbase.apache.org
> >Sent: Saturday, August 3, 2013 8:56 PM
> >
> >Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
> >
> >
> >On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Does this mean that user code wouldn't be able to depend
> >> on HConnectionManager for connection sharing ?
> >>
> >
> >My read of the above is that the HConnection instance is shared across
> >consumers, is the shared connection. Am I reading that correctly?
> >
> >On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Ah, I find the JIRA - HBASE-9117.
> >> >
> >> > Cheers
> >> >
> >> >
> >> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org>
> wrote:
> >> >
> >> >> Yeah, I filed a separate ticket for the API removal in trunk.
> >> >>
> >> >>
> >> >>
> >> >> ________________________________
> >> >> From: Ted Yu <yu...@gmail.com>
> >> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> >> >> Sent: Friday, August 2, 2013 10:31 PM
> >> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94,
> 0.95/0.96,
> >> >> and removed in 0.98
> >> >>
> >> >>
> >> >> bq. HConnectionManager.getConnection() will be removed.
> >> >>
> >> >> I don't see the above change in 6580-trunk.txt
> >> >> Would the above be done in next patch or in another JIRA ?
> >> >>
> >> >> Cheers
> >> >>
> >> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org>
> wrote:
> >> >>
> >> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >> >> >
> >> >> > The new proposed API looks like this:
> >> >> >
> >> >> > Here's the proposed new API:
> >> >> > * HConnectionManager:
> >> >> > public static HConnection createConnection(Configuration conf)
> >> >> > public static HConnection createConnection(Configuration conf,
> >> >> > ExecutorService pool)
> >> >> >
> >> >> > * HConnection:
> >> >> > public HTableInterface getTable(byte[] tableName) throws
> >> IOException
> >> >> > public HTableInterface getTable(byte[] tableName,
> ExecutorService
> >> >> > pool) throws IOException
> >> >> > public HTableInterface getTable(String tableName) throws
> >> IOException
> >> >> >
> >> >> > By default HConnectionImplementation will create an ExecutorService
> >> when
> >> >> > needed. The ExecutorService can optionally passed be passed in.
> >> >> > HTableInterfaces are retrieved from the HConnection. By default the
> >> >> > HConnection's ExecutorService is used, but optionally that can be
> >> >> > overridden for each HTable.
> >> >> >
> >> >> > In 0.98/trunk:
> >> >> >
> >> >> > 1. HTablePool will be removed. It is not longer needed.
> >> >> > 2. All constructors in HTable will be removed and changed to be
> >> >> protected.
> >> >> > All code use HTableInterface only.
> >> >> > 3. HConnectionManager.getConnection() will be removed.
> >> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
> >> removed,
> >> >> as
> >> >> > it is no longer needed.
> >> >> >
> >> >> >
> >> >> > The new flow of setting up a client would look like this:
> >> >> >
> >> >> > ----- Snip -----
> >> >> > // connection to the cluster
> >> >> > HConnection conn = HConnectionManager.createConnection(conf);
> >> >> > ...
> >> >> > // When the cluster connection is established get an
> HTableInterface
> >> for
> >> >> > each operation or thread.
> >> >> > // HConnection.getTable(...) is lightweight. The table is really
> just
> >> a
> >> >> > convenient place to call table method and for a temporary batch
> cache.
> >> >> > // It is in fact less overhead than HTablePool had when retrieving
> a
> >> >> > cached HTable.
> >> >> > // The HTableInterface returned is not thread safe as before.
> >> >> > // It's fine to get 1000's of these.
> >> >> > // Don't cache the longer than the lifetime of the HConnection
> >> >> > HTableInterface table = conn.getTable("MyTable");
> >> >> > ...
> >> >> > // just flushes outstanding commit, no futher cleanup needed, can
> be
> >> >> > omitted.
> >> >> > // HConnection holds no references to the returned HTable objects,
> >> they
> >> >> > can be GC'd as soon as they leave scope.
> >> >> > table.close();
> >> >> > ...
> >> >> > conn.close(); // done with the cluster, release resources
> >> >> > ----- Snip -----
> >> >> >
> >> >> > The HConnection will maintain and share its own ThreadPool for all
> >> batch
> >> >> > operations executed by the HTables.
> >> >> > This can overridden per HConnection and/or per individual HTable
> >> object.
> >> >> >
> >> >> > I will commit the new API to all branches early next week.
> >> >> >
> >> >> > Questions? Comments? Concerns? Praise?
> >> >> >
> >> >> > -- Lars
> >> >>
> >> >
> >> >
> >>
>
>
>
>
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
Posted by lars hofhansl <la...@apache.org>.
Let's do a little quiz:
HTable t1 = new HTable(conf);
t1.close();
// 1. Will the next line create a new HConnection behind the scenes (along with re-creating all the caches)?
// (If so, it will be expensive, if not, when is the first HConnection actually released?)
HTable t2 = new HTable(conf);
// 2. how about this one?
HTable t2 = new HTable(new Configuration(conf));
// 3. or now?
conf.setInt(HConstants.HBASE_CLIENT_PAUSE, 2000);
HTable t3 = new HTable(conf);
// 4. and now?
conf.setInt(HBASE_CLIENT_SCANNER_MAX_RESULT_SIZE_KEY, 1024000);
HTable t4 = new HTable(conf);
// 5. how many connections are opened now?
t4.close();
This stuff is convoluted and needlessly complicated. And this is not because the code is bad, but because the abstraction is simply inadequate.
A client wants to connect to a cluster and then do some action on that cluster (via HTable as a convenience).
If the cluster connection is implicit it leads to all of the above considerations.
(#1: Yes, #2: no, #3: yes, #4: no, #5: I don't really know, id'd have run it to see)
-- Lars
________________________________
From: Ted Yu <yu...@gmail.com>
To: lars hofhansl <la...@apache.org>
Cc: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Sunday, August 4, 2013 7:39 PM
Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
In the Connections "managing" HTables case, don't we need to figure out when an HConnection should be released ?
On Sun, Aug 4, 2013 at 7:23 PM, lars hofhansl <la...@apache.org> wrote:
Just look at HConnectionKey part, and hoops we go through to detect whether HConnections are the same or not, when to cache them, when/how to release them.
>In fact almost all HConnectionManager does is managing HConnections on behalf of HTable, when it should be other way around.
>
>Typically, when things get hard to explain (check out the comments in HConnectionManager) there is either an abstraction missing, or the abstraction is not right.
>The reverse (Connections "managing" HTables) has none of this.
>
>
>-- Lars
>
>
>_______________________________
>From: Ted Yu <yu...@gmail.com>
>To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>Sent: Sunday, August 4, 2013 4:27 PM
>
>Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
>
>
>
>bq. no funny business with unique Configurations
>
>Mind telling us what is funny about this part ?
>
>
>On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote:
>
>Correct. The HConnection is naturally shared between the HTables.
>>There is no longer any need to worry about this (no funny business with unique Configurations, in fact most of the code in HConnectionManager can be removed in trunk).
>>
>>It is also correct that the code now has to hold on the created HConnection, rather asking HConnectionManager for it.
>>
>>-- Lars
>>
>>
>>
>>________________________________
>> From: Nick Dimiduk <nd...@gmail.com>
>>To: dev@hbase.apache.org
>>Sent: Saturday, August 3, 2013 8:56 PM
>>
>>Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
>>
>>
>>On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Does this mean that user code wouldn't be able to depend
>>> on HConnectionManager for connection sharing ?
>>>
>>
>>My read of the above is that the HConnection instance is shared across
>>consumers, is the shared connection. Am I reading that correctly?
>>
>>On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> > Ah, I find the JIRA - HBASE-9117.
>>> >
>>> > Cheers
>>> >
>>> >
>>> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> wrote:
>>> >
>>> >> Yeah, I filed a separate ticket for the API removal in trunk.
>>> >>
>>> >>
>>> >>
>>> >> ________________________________
>>> >> From: Ted Yu <yu...@gmail.com>
>>> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>>> >> Sent: Friday, August 2, 2013 10:31 PM
>>> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
>>> >> and removed in 0.98
>>> >>
>>> >>
>>> >> bq. HConnectionManager.getConnection() will be removed.
>>> >>
>>> >> I don't see the above change in 6580-trunk.txt
>>> >> Would the above be done in next patch or in another JIRA ?
>>> >>
>>> >> Cheers
>>> >>
>>> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
>>> >>
>>> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
>>> >> >
>>> >> > The new proposed API looks like this:
>>> >> >
>>> >> > Here's the proposed new API:
>>> >> > * HConnectionManager:
>>> >> > public static HConnection createConnection(Configuration conf)
>>> >> > public static HConnection createConnection(Configuration conf,
>>> >> > ExecutorService pool)
>>> >> >
>>> >> > * HConnection:
>>> >> > public HTableInterface getTable(byte[] tableName) throws
>>> IOException
>>> >> > public HTableInterface getTable(byte[] tableName, ExecutorService
>>> >> > pool) throws IOException
>>> >> > public HTableInterface getTable(String tableName) throws
>>> IOException
>>> >> >
>>> >> > By default HConnectionImplementation will create an ExecutorService
>>> when
>>> >> > needed. The ExecutorService can optionally passed be passed in.
>>> >> > HTableInterfaces are retrieved from the HConnection. By default the
>>> >> > HConnection's ExecutorService is used, but optionally that can be
>>> >> > overridden for each HTable.
>>> >> >
>>> >> > In 0.98/trunk:
>>> >> >
>>> >> > 1. HTablePool will be removed. It is not longer needed.
>>> >> > 2. All constructors in HTable will be removed and changed to be
>>> >> protected.
>>> >> > All code use HTableInterface only.
>>> >> > 3. HConnectionManager.getConnection() will be removed.
>>> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
>>> removed,
>>> >> as
>>> >> > it is no longer needed.
>>> >> >
>>> >> >
>>> >> > The new flow of setting up a client would look like this:
>>> >> >
>>> >> > ----- Snip -----
>>> >> > // connection to the cluster
>>> >> > HConnection conn = HConnectionManager.createConnection(conf);
>>> >> > ...
>>> >> > // When the cluster connection is established get an HTableInterface
>>> for
>>> >> > each operation or thread.
>>> >> > // HConnection.getTable(...) is lightweight. The table is really just
>>> a
>>> >> > convenient place to call table method and for a temporary batch cache.
>>> >> > // It is in fact less overhead than HTablePool had when retrieving a
>>> >> > cached HTable.
>>> >> > // The HTableInterface returned is not thread safe as before.
>>> >> > // It's fine to get 1000's of these.
>>> >> > // Don't cache the longer than the lifetime of the HConnection
>>> >> > HTableInterface table = conn.getTable("MyTable");
>>> >> > ...
>>> >> > // just flushes outstanding commit, no futher cleanup needed, can be
>>> >> > omitted.
>>> >> > // HConnection holds no references to the returned HTable objects,
>>> they
>>> >> > can be GC'd as soon as they leave scope.
>>> >> > table.close();
>>> >> > ...
>>> >> > conn.close(); // done with the cluster, release resources
>>> >> > ----- Snip -----
>>> >> >
>>> >> > The HConnection will maintain and share its own ThreadPool for all
>>> batch
>>> >> > operations executed by the HTables.
>>> >> > This can overridden per HConnection and/or per individual HTable
>>> object.
>>> >> >
>>> >> > I will commit the new API to all branches early next week.
>>> >> >
>>> >> > Questions? Comments? Concerns? Praise?
>>> >> >
>>> >> > -- Lars
>>> >>
>>> >
>>> >
>>>
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
In the Connections "managing" HTables case, don't we need to figure out
when an HConnection should be released ?
On Sun, Aug 4, 2013 at 7:23 PM, lars hofhansl <la...@apache.org> wrote:
> Just look at HConnectionKey part, and hoops we go through to detect
> whether HConnections are the same or not, when to cache them, when/how to
> release them.
> In fact almost all HConnectionManager does is managing HConnections on
> behalf of HTable, when it should be other way around.
>
> Typically, when things get hard to explain (check out the comments in
> HConnectionManager) there is either an abstraction missing, or the
> abstraction is not right.
> The reverse (Connections "managing" HTables) has none of this.
>
>
> -- Lars
>
> _______________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Sunday, August 4, 2013 4:27 PM
> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
>
>
>
> bq. no funny business with unique Configurations
>
> Mind telling us what is funny about this part ?
>
>
> On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote:
>
> Correct. The HConnection is naturally shared between the HTables.
> >There is no longer any need to worry about this (no funny business with
> unique Configurations, in fact most of the code in HConnectionManager can
> be removed in trunk).
> >
> >It is also correct that the code now has to hold on the created
> HConnection, rather asking HConnectionManager for it.
> >
> >-- Lars
> >
> >
> >
> >________________________________
> > From: Nick Dimiduk <nd...@gmail.com>
> >To: dev@hbase.apache.org
> >Sent: Saturday, August 3, 2013 8:56 PM
> >
> >Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
> >
> >
> >On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Does this mean that user code wouldn't be able to depend
> >> on HConnectionManager for connection sharing ?
> >>
> >
> >My read of the above is that the HConnection instance is shared across
> >consumers, is the shared connection. Am I reading that correctly?
> >
> >On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > Ah, I find the JIRA - HBASE-9117.
> >> >
> >> > Cheers
> >> >
> >> >
> >> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org>
> wrote:
> >> >
> >> >> Yeah, I filed a separate ticket for the API removal in trunk.
> >> >>
> >> >>
> >> >>
> >> >> ________________________________
> >> >> From: Ted Yu <yu...@gmail.com>
> >> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> >> >> Sent: Friday, August 2, 2013 10:31 PM
> >> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94,
> 0.95/0.96,
> >> >> and removed in 0.98
> >> >>
> >> >>
> >> >> bq. HConnectionManager.getConnection() will be removed.
> >> >>
> >> >> I don't see the above change in 6580-trunk.txt
> >> >> Would the above be done in next patch or in another JIRA ?
> >> >>
> >> >> Cheers
> >> >>
> >> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org>
> wrote:
> >> >>
> >> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >> >> >
> >> >> > The new proposed API looks like this:
> >> >> >
> >> >> > Here's the proposed new API:
> >> >> > * HConnectionManager:
> >> >> > public static HConnection createConnection(Configuration conf)
> >> >> > public static HConnection createConnection(Configuration conf,
> >> >> > ExecutorService pool)
> >> >> >
> >> >> > * HConnection:
> >> >> > public HTableInterface getTable(byte[] tableName) throws
> >> IOException
> >> >> > public HTableInterface getTable(byte[] tableName,
> ExecutorService
> >> >> > pool) throws IOException
> >> >> > public HTableInterface getTable(String tableName) throws
> >> IOException
> >> >> >
> >> >> > By default HConnectionImplementation will create an ExecutorService
> >> when
> >> >> > needed. The ExecutorService can optionally passed be passed in.
> >> >> > HTableInterfaces are retrieved from the HConnection. By default the
> >> >> > HConnection's ExecutorService is used, but optionally that can be
> >> >> > overridden for each HTable.
> >> >> >
> >> >> > In 0.98/trunk:
> >> >> >
> >> >> > 1. HTablePool will be removed. It is not longer needed.
> >> >> > 2. All constructors in HTable will be removed and changed to be
> >> >> protected.
> >> >> > All code use HTableInterface only.
> >> >> > 3. HConnectionManager.getConnection() will be removed.
> >> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
> >> removed,
> >> >> as
> >> >> > it is no longer needed.
> >> >> >
> >> >> >
> >> >> > The new flow of setting up a client would look like this:
> >> >> >
> >> >> > ----- Snip -----
> >> >> > // connection to the cluster
> >> >> > HConnection conn = HConnectionManager.createConnection(conf);
> >> >> > ...
> >> >> > // When the cluster connection is established get an
> HTableInterface
> >> for
> >> >> > each operation or thread.
> >> >> > // HConnection.getTable(...) is lightweight. The table is really
> just
> >> a
> >> >> > convenient place to call table method and for a temporary batch
> cache.
> >> >> > // It is in fact less overhead than HTablePool had when retrieving
> a
> >> >> > cached HTable.
> >> >> > // The HTableInterface returned is not thread safe as before.
> >> >> > // It's fine to get 1000's of these.
> >> >> > // Don't cache the longer than the lifetime of the HConnection
> >> >> > HTableInterface table = conn.getTable("MyTable");
> >> >> > ...
> >> >> > // just flushes outstanding commit, no futher cleanup needed, can
> be
> >> >> > omitted.
> >> >> > // HConnection holds no references to the returned HTable objects,
> >> they
> >> >> > can be GC'd as soon as they leave scope.
> >> >> > table.close();
> >> >> > ...
> >> >> > conn.close(); // done with the cluster, release resources
> >> >> > ----- Snip -----
> >> >> >
> >> >> > The HConnection will maintain and share its own ThreadPool for all
> >> batch
> >> >> > operations executed by the HTables.
> >> >> > This can overridden per HConnection and/or per individual HTable
> >> object.
> >> >> >
> >> >> > I will commit the new API to all branches early next week.
> >> >> >
> >> >> > Questions? Comments? Concerns? Praise?
> >> >> >
> >> >> > -- Lars
> >> >>
> >> >
> >> >
> >>
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
Posted by lars hofhansl <la...@apache.org>.
Just look at HConnectionKey part, and hoops we go through to detect whether HConnections are the same or not, when to cache them, when/how to release them.
In fact almost all HConnectionManager does is managing HConnections on behalf of HTable, when it should be other way around.
Typically, when things get hard to explain (check out the comments in HConnectionManager) there is either an abstraction missing, or the abstraction is not right.
The reverse (Connections "managing" HTables) has none of this.
-- Lars
_______________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Sunday, August 4, 2013 4:27 PM
Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
bq. no funny business with unique Configurations
Mind telling us what is funny about this part ?
On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote:
Correct. The HConnection is naturally shared between the HTables.
>There is no longer any need to worry about this (no funny business with unique Configurations, in fact most of the code in HConnectionManager can be removed in trunk).
>
>It is also correct that the code now has to hold on the created HConnection, rather asking HConnectionManager for it.
>
>-- Lars
>
>
>
>________________________________
> From: Nick Dimiduk <nd...@gmail.com>
>To: dev@hbase.apache.org
>Sent: Saturday, August 3, 2013 8:56 PM
>
>Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
>
>
>On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Does this mean that user code wouldn't be able to depend
>> on HConnectionManager for connection sharing ?
>>
>
>My read of the above is that the HConnection instance is shared across
>consumers, is the shared connection. Am I reading that correctly?
>
>On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > Ah, I find the JIRA - HBASE-9117.
>> >
>> > Cheers
>> >
>> >
>> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> wrote:
>> >
>> >> Yeah, I filed a separate ticket for the API removal in trunk.
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> From: Ted Yu <yu...@gmail.com>
>> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> >> Sent: Friday, August 2, 2013 10:31 PM
>> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
>> >> and removed in 0.98
>> >>
>> >>
>> >> bq. HConnectionManager.getConnection() will be removed.
>> >>
>> >> I don't see the above change in 6580-trunk.txt
>> >> Would the above be done in next patch or in another JIRA ?
>> >>
>> >> Cheers
>> >>
>> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
>> >>
>> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
>> >> >
>> >> > The new proposed API looks like this:
>> >> >
>> >> > Here's the proposed new API:
>> >> > * HConnectionManager:
>> >> > public static HConnection createConnection(Configuration conf)
>> >> > public static HConnection createConnection(Configuration conf,
>> >> > ExecutorService pool)
>> >> >
>> >> > * HConnection:
>> >> > public HTableInterface getTable(byte[] tableName) throws
>> IOException
>> >> > public HTableInterface getTable(byte[] tableName, ExecutorService
>> >> > pool) throws IOException
>> >> > public HTableInterface getTable(String tableName) throws
>> IOException
>> >> >
>> >> > By default HConnectionImplementation will create an ExecutorService
>> when
>> >> > needed. The ExecutorService can optionally passed be passed in.
>> >> > HTableInterfaces are retrieved from the HConnection. By default the
>> >> > HConnection's ExecutorService is used, but optionally that can be
>> >> > overridden for each HTable.
>> >> >
>> >> > In 0.98/trunk:
>> >> >
>> >> > 1. HTablePool will be removed. It is not longer needed.
>> >> > 2. All constructors in HTable will be removed and changed to be
>> >> protected.
>> >> > All code use HTableInterface only.
>> >> > 3. HConnectionManager.getConnection() will be removed.
>> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
>> removed,
>> >> as
>> >> > it is no longer needed.
>> >> >
>> >> >
>> >> > The new flow of setting up a client would look like this:
>> >> >
>> >> > ----- Snip -----
>> >> > // connection to the cluster
>> >> > HConnection conn = HConnectionManager.createConnection(conf);
>> >> > ...
>> >> > // When the cluster connection is established get an HTableInterface
>> for
>> >> > each operation or thread.
>> >> > // HConnection.getTable(...) is lightweight. The table is really just
>> a
>> >> > convenient place to call table method and for a temporary batch cache.
>> >> > // It is in fact less overhead than HTablePool had when retrieving a
>> >> > cached HTable.
>> >> > // The HTableInterface returned is not thread safe as before.
>> >> > // It's fine to get 1000's of these.
>> >> > // Don't cache the longer than the lifetime of the HConnection
>> >> > HTableInterface table = conn.getTable("MyTable");
>> >> > ...
>> >> > // just flushes outstanding commit, no futher cleanup needed, can be
>> >> > omitted.
>> >> > // HConnection holds no references to the returned HTable objects,
>> they
>> >> > can be GC'd as soon as they leave scope.
>> >> > table.close();
>> >> > ...
>> >> > conn.close(); // done with the cluster, release resources
>> >> > ----- Snip -----
>> >> >
>> >> > The HConnection will maintain and share its own ThreadPool for all
>> batch
>> >> > operations executed by the HTables.
>> >> > This can overridden per HConnection and/or per individual HTable
>> object.
>> >> >
>> >> > I will commit the new API to all branches early next week.
>> >> >
>> >> > Questions? Comments? Concerns? Praise?
>> >> >
>> >> > -- Lars
>> >>
>> >
>> >
>>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
bq. no funny business with unique Configurations
Mind telling us what is funny about this part ?
On Sat, Aug 3, 2013 at 10:41 PM, lars hofhansl <la...@apache.org> wrote:
> Correct. The HConnection is naturally shared between the HTables.
> There is no longer any need to worry about this (no funny business with
> unique Configurations, in fact most of the code in HConnectionManager can
> be removed in trunk).
>
> It is also correct that the code now has to hold on the created
> HConnection, rather asking HConnectionManager for it.
>
> -- Lars
>
>
>
> ________________________________
> From: Nick Dimiduk <nd...@gmail.com>
> To: dev@hbase.apache.org
> Sent: Saturday, August 3, 2013 8:56 PM
> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
>
>
> On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Does this mean that user code wouldn't be able to depend
> > on HConnectionManager for connection sharing ?
> >
>
> My read of the above is that the HConnection instance is shared across
> consumers, is the shared connection. Am I reading that correctly?
>
> On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Ah, I find the JIRA - HBASE-9117.
> > >
> > > Cheers
> > >
> > >
> > > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org>
> wrote:
> > >
> > >> Yeah, I filed a separate ticket for the API removal in trunk.
> > >>
> > >>
> > >>
> > >> ________________________________
> > >> From: Ted Yu <yu...@gmail.com>
> > >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> > >> Sent: Friday, August 2, 2013 10:31 PM
> > >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94,
> 0.95/0.96,
> > >> and removed in 0.98
> > >>
> > >>
> > >> bq. HConnectionManager.getConnection() will be removed.
> > >>
> > >> I don't see the above change in 6580-trunk.txt
> > >> Would the above be done in next patch or in another JIRA ?
> > >>
> > >> Cheers
> > >>
> > >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org>
> wrote:
> > >>
> > >> > See. https://issues.apache.org/jira/browse/HBASE-6580
> > >> >
> > >> > The new proposed API looks like this:
> > >> >
> > >> > Here's the proposed new API:
> > >> > * HConnectionManager:
> > >> > public static HConnection createConnection(Configuration conf)
> > >> > public static HConnection createConnection(Configuration conf,
> > >> > ExecutorService pool)
> > >> >
> > >> > * HConnection:
> > >> > public HTableInterface getTable(byte[] tableName) throws
> > IOException
> > >> > public HTableInterface getTable(byte[] tableName,
> ExecutorService
> > >> > pool) throws IOException
> > >> > public HTableInterface getTable(String tableName) throws
> > IOException
> > >> >
> > >> > By default HConnectionImplementation will create an ExecutorService
> > when
> > >> > needed. The ExecutorService can optionally passed be passed in.
> > >> > HTableInterfaces are retrieved from the HConnection. By default the
> > >> > HConnection's ExecutorService is used, but optionally that can be
> > >> > overridden for each HTable.
> > >> >
> > >> > In 0.98/trunk:
> > >> >
> > >> > 1. HTablePool will be removed. It is not longer needed.
> > >> > 2. All constructors in HTable will be removed and changed to be
> > >> protected.
> > >> > All code use HTableInterface only.
> > >> > 3. HConnectionManager.getConnection() will be removed.
> > >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
> > removed,
> > >> as
> > >> > it is no longer needed.
> > >> >
> > >> >
> > >> > The new flow of setting up a client would look like this:
> > >> >
> > >> > ----- Snip -----
> > >> > // connection to the cluster
> > >> > HConnection conn = HConnectionManager.createConnection(conf);
> > >> > ...
> > >> > // When the cluster connection is established get an HTableInterface
> > for
> > >> > each operation or thread.
> > >> > // HConnection.getTable(...) is lightweight. The table is really
> just
> > a
> > >> > convenient place to call table method and for a temporary batch
> cache.
> > >> > // It is in fact less overhead than HTablePool had when retrieving a
> > >> > cached HTable.
> > >> > // The HTableInterface returned is not thread safe as before.
> > >> > // It's fine to get 1000's of these.
> > >> > // Don't cache the longer than the lifetime of the HConnection
> > >> > HTableInterface table = conn.getTable("MyTable");
> > >> > ...
> > >> > // just flushes outstanding commit, no futher cleanup needed, can be
> > >> > omitted.
> > >> > // HConnection holds no references to the returned HTable objects,
> > they
> > >> > can be GC'd as soon as they leave scope.
> > >> > table.close();
> > >> > ...
> > >> > conn.close(); // done with the cluster, release resources
> > >> > ----- Snip -----
> > >> >
> > >> > The HConnection will maintain and share its own ThreadPool for all
> > batch
> > >> > operations executed by the HTables.
> > >> > This can overridden per HConnection and/or per individual HTable
> > object.
> > >> >
> > >> > I will commit the new API to all branches early next week.
> > >> >
> > >> > Questions? Comments? Concerns? Praise?
> > >> >
> > >> > -- Lars
> > >>
> > >
> > >
> >
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
Posted by lars hofhansl <la...@apache.org>.
Correct. The HConnection is naturally shared between the HTables.
There is no longer any need to worry about this (no funny business with unique Configurations, in fact most of the code in HConnectionManager can be removed in trunk).
It is also correct that the code now has to hold on the created HConnection, rather asking HConnectionManager for it.
-- Lars
________________________________
From: Nick Dimiduk <nd...@gmail.com>
To: dev@hbase.apache.org
Sent: Saturday, August 3, 2013 8:56 PM
Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
> Does this mean that user code wouldn't be able to depend
> on HConnectionManager for connection sharing ?
>
My read of the above is that the HConnection instance is shared across
consumers, is the shared connection. Am I reading that correctly?
On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Ah, I find the JIRA - HBASE-9117.
> >
> > Cheers
> >
> >
> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> wrote:
> >
> >> Yeah, I filed a separate ticket for the API removal in trunk.
> >>
> >>
> >>
> >> ________________________________
> >> From: Ted Yu <yu...@gmail.com>
> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> >> Sent: Friday, August 2, 2013 10:31 PM
> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> >> and removed in 0.98
> >>
> >>
> >> bq. HConnectionManager.getConnection() will be removed.
> >>
> >> I don't see the above change in 6580-trunk.txt
> >> Would the above be done in next patch or in another JIRA ?
> >>
> >> Cheers
> >>
> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
> >>
> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >> >
> >> > The new proposed API looks like this:
> >> >
> >> > Here's the proposed new API:
> >> > * HConnectionManager:
> >> > public static HConnection createConnection(Configuration conf)
> >> > public static HConnection createConnection(Configuration conf,
> >> > ExecutorService pool)
> >> >
> >> > * HConnection:
> >> > public HTableInterface getTable(byte[] tableName) throws
> IOException
> >> > public HTableInterface getTable(byte[] tableName, ExecutorService
> >> > pool) throws IOException
> >> > public HTableInterface getTable(String tableName) throws
> IOException
> >> >
> >> > By default HConnectionImplementation will create an ExecutorService
> when
> >> > needed. The ExecutorService can optionally passed be passed in.
> >> > HTableInterfaces are retrieved from the HConnection. By default the
> >> > HConnection's ExecutorService is used, but optionally that can be
> >> > overridden for each HTable.
> >> >
> >> > In 0.98/trunk:
> >> >
> >> > 1. HTablePool will be removed. It is not longer needed.
> >> > 2. All constructors in HTable will be removed and changed to be
> >> protected.
> >> > All code use HTableInterface only.
> >> > 3. HConnectionManager.getConnection() will be removed.
> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
> removed,
> >> as
> >> > it is no longer needed.
> >> >
> >> >
> >> > The new flow of setting up a client would look like this:
> >> >
> >> > ----- Snip -----
> >> > // connection to the cluster
> >> > HConnection conn = HConnectionManager.createConnection(conf);
> >> > ...
> >> > // When the cluster connection is established get an HTableInterface
> for
> >> > each operation or thread.
> >> > // HConnection.getTable(...) is lightweight. The table is really just
> a
> >> > convenient place to call table method and for a temporary batch cache.
> >> > // It is in fact less overhead than HTablePool had when retrieving a
> >> > cached HTable.
> >> > // The HTableInterface returned is not thread safe as before.
> >> > // It's fine to get 1000's of these.
> >> > // Don't cache the longer than the lifetime of the HConnection
> >> > HTableInterface table = conn.getTable("MyTable");
> >> > ...
> >> > // just flushes outstanding commit, no futher cleanup needed, can be
> >> > omitted.
> >> > // HConnection holds no references to the returned HTable objects,
> they
> >> > can be GC'd as soon as they leave scope.
> >> > table.close();
> >> > ...
> >> > conn.close(); // done with the cluster, release resources
> >> > ----- Snip -----
> >> >
> >> > The HConnection will maintain and share its own ThreadPool for all
> batch
> >> > operations executed by the HTables.
> >> > This can overridden per HConnection and/or per individual HTable
> object.
> >> >
> >> > I will commit the new API to all branches early next week.
> >> >
> >> > Questions? Comments? Concerns? Praise?
> >> >
> >> > -- Lars
> >>
> >
> >
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Nick Dimiduk <nd...@gmail.com>.
On Sat, Aug 3, 2013 at 8:52 PM, Ted Yu <yu...@gmail.com> wrote:
> Does this mean that user code wouldn't be able to depend
> on HConnectionManager for connection sharing ?
>
My read of the above is that the HConnection instance is shared across
consumers, is the shared connection. Am I reading that correctly?
On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Ah, I find the JIRA - HBASE-9117.
> >
> > Cheers
> >
> >
> > On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> wrote:
> >
> >> Yeah, I filed a separate ticket for the API removal in trunk.
> >>
> >>
> >>
> >> ________________________________
> >> From: Ted Yu <yu...@gmail.com>
> >> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> >> Sent: Friday, August 2, 2013 10:31 PM
> >> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> >> and removed in 0.98
> >>
> >>
> >> bq. HConnectionManager.getConnection() will be removed.
> >>
> >> I don't see the above change in 6580-trunk.txt
> >> Would the above be done in next patch or in another JIRA ?
> >>
> >> Cheers
> >>
> >> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
> >>
> >> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >> >
> >> > The new proposed API looks like this:
> >> >
> >> > Here's the proposed new API:
> >> > * HConnectionManager:
> >> > public static HConnection createConnection(Configuration conf)
> >> > public static HConnection createConnection(Configuration conf,
> >> > ExecutorService pool)
> >> >
> >> > * HConnection:
> >> > public HTableInterface getTable(byte[] tableName) throws
> IOException
> >> > public HTableInterface getTable(byte[] tableName, ExecutorService
> >> > pool) throws IOException
> >> > public HTableInterface getTable(String tableName) throws
> IOException
> >> >
> >> > By default HConnectionImplementation will create an ExecutorService
> when
> >> > needed. The ExecutorService can optionally passed be passed in.
> >> > HTableInterfaces are retrieved from the HConnection. By default the
> >> > HConnection's ExecutorService is used, but optionally that can be
> >> > overridden for each HTable.
> >> >
> >> > In 0.98/trunk:
> >> >
> >> > 1. HTablePool will be removed. It is not longer needed.
> >> > 2. All constructors in HTable will be removed and changed to be
> >> protected.
> >> > All code use HTableInterface only.
> >> > 3. HConnectionManager.getConnection() will be removed.
> >> > 3. All HConnection caching (deleteConnection, etc,etc) will be
> removed,
> >> as
> >> > it is no longer needed.
> >> >
> >> >
> >> > The new flow of setting up a client would look like this:
> >> >
> >> > ----- Snip -----
> >> > // connection to the cluster
> >> > HConnection conn = HConnectionManager.createConnection(conf);
> >> > ...
> >> > // When the cluster connection is established get an HTableInterface
> for
> >> > each operation or thread.
> >> > // HConnection.getTable(...) is lightweight. The table is really just
> a
> >> > convenient place to call table method and for a temporary batch cache.
> >> > // It is in fact less overhead than HTablePool had when retrieving a
> >> > cached HTable.
> >> > // The HTableInterface returned is not thread safe as before.
> >> > // It's fine to get 1000's of these.
> >> > // Don't cache the longer than the lifetime of the HConnection
> >> > HTableInterface table = conn.getTable("MyTable");
> >> > ...
> >> > // just flushes outstanding commit, no futher cleanup needed, can be
> >> > omitted.
> >> > // HConnection holds no references to the returned HTable objects,
> they
> >> > can be GC'd as soon as they leave scope.
> >> > table.close();
> >> > ...
> >> > conn.close(); // done with the cluster, release resources
> >> > ----- Snip -----
> >> >
> >> > The HConnection will maintain and share its own ThreadPool for all
> batch
> >> > operations executed by the HTables.
> >> > This can overridden per HConnection and/or per individual HTable
> object.
> >> >
> >> > I will commit the new API to all branches early next week.
> >> >
> >> > Questions? Comments? Concerns? Praise?
> >> >
> >> > -- Lars
> >>
> >
> >
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
Does this mean that user code wouldn't be able to depend
on HConnectionManager for connection sharing ?
If so, user may need to develop his / her own code for connection sharing ?
Cheers
On Sat, Aug 3, 2013 at 7:20 AM, Ted Yu <yu...@gmail.com> wrote:
> Ah, I find the JIRA - HBASE-9117.
>
> Cheers
>
>
> On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> wrote:
>
>> Yeah, I filed a separate ticket for the API removal in trunk.
>>
>>
>>
>> ________________________________
>> From: Ted Yu <yu...@gmail.com>
>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> Sent: Friday, August 2, 2013 10:31 PM
>> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
>> and removed in 0.98
>>
>>
>> bq. HConnectionManager.getConnection() will be removed.
>>
>> I don't see the above change in 6580-trunk.txt
>> Would the above be done in next patch or in another JIRA ?
>>
>> Cheers
>>
>> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
>>
>> > See. https://issues.apache.org/jira/browse/HBASE-6580
>> >
>> > The new proposed API looks like this:
>> >
>> > Here's the proposed new API:
>> > * HConnectionManager:
>> > public static HConnection createConnection(Configuration conf)
>> > public static HConnection createConnection(Configuration conf,
>> > ExecutorService pool)
>> >
>> > * HConnection:
>> > public HTableInterface getTable(byte[] tableName) throws IOException
>> > public HTableInterface getTable(byte[] tableName, ExecutorService
>> > pool) throws IOException
>> > public HTableInterface getTable(String tableName) throws IOException
>> >
>> > By default HConnectionImplementation will create an ExecutorService when
>> > needed. The ExecutorService can optionally passed be passed in.
>> > HTableInterfaces are retrieved from the HConnection. By default the
>> > HConnection's ExecutorService is used, but optionally that can be
>> > overridden for each HTable.
>> >
>> > In 0.98/trunk:
>> >
>> > 1. HTablePool will be removed. It is not longer needed.
>> > 2. All constructors in HTable will be removed and changed to be
>> protected.
>> > All code use HTableInterface only.
>> > 3. HConnectionManager.getConnection() will be removed.
>> > 3. All HConnection caching (deleteConnection, etc,etc) will be removed,
>> as
>> > it is no longer needed.
>> >
>> >
>> > The new flow of setting up a client would look like this:
>> >
>> > ----- Snip -----
>> > // connection to the cluster
>> > HConnection conn = HConnectionManager.createConnection(conf);
>> > ...
>> > // When the cluster connection is established get an HTableInterface for
>> > each operation or thread.
>> > // HConnection.getTable(...) is lightweight. The table is really just a
>> > convenient place to call table method and for a temporary batch cache.
>> > // It is in fact less overhead than HTablePool had when retrieving a
>> > cached HTable.
>> > // The HTableInterface returned is not thread safe as before.
>> > // It's fine to get 1000's of these.
>> > // Don't cache the longer than the lifetime of the HConnection
>> > HTableInterface table = conn.getTable("MyTable");
>> > ...
>> > // just flushes outstanding commit, no futher cleanup needed, can be
>> > omitted.
>> > // HConnection holds no references to the returned HTable objects, they
>> > can be GC'd as soon as they leave scope.
>> > table.close();
>> > ...
>> > conn.close(); // done with the cluster, release resources
>> > ----- Snip -----
>> >
>> > The HConnection will maintain and share its own ThreadPool for all batch
>> > operations executed by the HTables.
>> > This can overridden per HConnection and/or per individual HTable object.
>> >
>> > I will commit the new API to all branches early next week.
>> >
>> > Questions? Comments? Concerns? Praise?
>> >
>> > -- Lars
>>
>
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
Ah, I find the JIRA - HBASE-9117.
Cheers
On Fri, Aug 2, 2013 at 10:54 PM, lars hofhansl <la...@apache.org> wrote:
> Yeah, I filed a separate ticket for the API removal in trunk.
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Friday, August 2, 2013 10:31 PM
> Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96,
> and removed in 0.98
>
>
> bq. HConnectionManager.getConnection() will be removed.
>
> I don't see the above change in 6580-trunk.txt
> Would the above be done in next patch or in another JIRA ?
>
> Cheers
>
> On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
>
> > See. https://issues.apache.org/jira/browse/HBASE-6580
> >
> > The new proposed API looks like this:
> >
> > Here's the proposed new API:
> > * HConnectionManager:
> > public static HConnection createConnection(Configuration conf)
> > public static HConnection createConnection(Configuration conf,
> > ExecutorService pool)
> >
> > * HConnection:
> > public HTableInterface getTable(byte[] tableName) throws IOException
> > public HTableInterface getTable(byte[] tableName, ExecutorService
> > pool) throws IOException
> > public HTableInterface getTable(String tableName) throws IOException
> >
> > By default HConnectionImplementation will create an ExecutorService when
> > needed. The ExecutorService can optionally passed be passed in.
> > HTableInterfaces are retrieved from the HConnection. By default the
> > HConnection's ExecutorService is used, but optionally that can be
> > overridden for each HTable.
> >
> > In 0.98/trunk:
> >
> > 1. HTablePool will be removed. It is not longer needed.
> > 2. All constructors in HTable will be removed and changed to be
> protected.
> > All code use HTableInterface only.
> > 3. HConnectionManager.getConnection() will be removed.
> > 3. All HConnection caching (deleteConnection, etc,etc) will be removed,
> as
> > it is no longer needed.
> >
> >
> > The new flow of setting up a client would look like this:
> >
> > ----- Snip -----
> > // connection to the cluster
> > HConnection conn = HConnectionManager.createConnection(conf);
> > ...
> > // When the cluster connection is established get an HTableInterface for
> > each operation or thread.
> > // HConnection.getTable(...) is lightweight. The table is really just a
> > convenient place to call table method and for a temporary batch cache.
> > // It is in fact less overhead than HTablePool had when retrieving a
> > cached HTable.
> > // The HTableInterface returned is not thread safe as before.
> > // It's fine to get 1000's of these.
> > // Don't cache the longer than the lifetime of the HConnection
> > HTableInterface table = conn.getTable("MyTable");
> > ...
> > // just flushes outstanding commit, no futher cleanup needed, can be
> > omitted.
> > // HConnection holds no references to the returned HTable objects, they
> > can be GC'd as soon as they leave scope.
> > table.close();
> > ...
> > conn.close(); // done with the cluster, release resources
> > ----- Snip -----
> >
> > The HConnection will maintain and share its own ThreadPool for all batch
> > operations executed by the HTables.
> > This can overridden per HConnection and/or per individual HTable object.
> >
> > I will commit the new API to all branches early next week.
> >
> > Questions? Comments? Concerns? Praise?
> >
> > -- Lars
>
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
Posted by lars hofhansl <la...@apache.org>.
Yeah, I filed a separate ticket for the API removal in trunk.
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Friday, August 2, 2013 10:31 PM
Subject: Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and removed in 0.98
bq. HConnectionManager.getConnection() will be removed.
I don't see the above change in 6580-trunk.txt
Would the above be done in next patch or in another JIRA ?
Cheers
On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
> See. https://issues.apache.org/jira/browse/HBASE-6580
>
> The new proposed API looks like this:
>
> Here's the proposed new API:
> * HConnectionManager:
> public static HConnection createConnection(Configuration conf)
> public static HConnection createConnection(Configuration conf,
> ExecutorService pool)
>
> * HConnection:
> public HTableInterface getTable(byte[] tableName) throws IOException
> public HTableInterface getTable(byte[] tableName, ExecutorService
> pool) throws IOException
> public HTableInterface getTable(String tableName) throws IOException
>
> By default HConnectionImplementation will create an ExecutorService when
> needed. The ExecutorService can optionally passed be passed in.
> HTableInterfaces are retrieved from the HConnection. By default the
> HConnection's ExecutorService is used, but optionally that can be
> overridden for each HTable.
>
> In 0.98/trunk:
>
> 1. HTablePool will be removed. It is not longer needed.
> 2. All constructors in HTable will be removed and changed to be protected.
> All code use HTableInterface only.
> 3. HConnectionManager.getConnection() will be removed.
> 3. All HConnection caching (deleteConnection, etc,etc) will be removed, as
> it is no longer needed.
>
>
> The new flow of setting up a client would look like this:
>
> ----- Snip -----
> // connection to the cluster
> HConnection conn = HConnectionManager.createConnection(conf);
> ...
> // When the cluster connection is established get an HTableInterface for
> each operation or thread.
> // HConnection.getTable(...) is lightweight. The table is really just a
> convenient place to call table method and for a temporary batch cache.
> // It is in fact less overhead than HTablePool had when retrieving a
> cached HTable.
> // The HTableInterface returned is not thread safe as before.
> // It's fine to get 1000's of these.
> // Don't cache the longer than the lifetime of the HConnection
> HTableInterface table = conn.getTable("MyTable");
> ...
> // just flushes outstanding commit, no futher cleanup needed, can be
> omitted.
> // HConnection holds no references to the returned HTable objects, they
> can be GC'd as soon as they leave scope.
> table.close();
> ...
> conn.close(); // done with the cluster, release resources
> ----- Snip -----
>
> The HConnection will maintain and share its own ThreadPool for all batch
> operations executed by the HTables.
> This can overridden per HConnection and/or per individual HTable object.
>
> I will commit the new API to all branches early next week.
>
> Questions? Comments? Concerns? Praise?
>
> -- Lars
Re: Heads up, HTablePool will be deprecated in 0.94, 0.95/0.96, and
removed in 0.98
Posted by Ted Yu <yu...@gmail.com>.
bq. HConnectionManager.getConnection() will be removed.
I don't see the above change in 6580-trunk.txt
Would the above be done in next patch or in another JIRA ?
Cheers
On Fri, Aug 2, 2013 at 9:29 PM, lars hofhansl <la...@apache.org> wrote:
> See. https://issues.apache.org/jira/browse/HBASE-6580
>
> The new proposed API looks like this:
>
> Here's the proposed new API:
> * HConnectionManager:
> public static HConnection createConnection(Configuration conf)
> public static HConnection createConnection(Configuration conf,
> ExecutorService pool)
>
> * HConnection:
> public HTableInterface getTable(byte[] tableName) throws IOException
> public HTableInterface getTable(byte[] tableName, ExecutorService
> pool) throws IOException
> public HTableInterface getTable(String tableName) throws IOException
>
> By default HConnectionImplementation will create an ExecutorService when
> needed. The ExecutorService can optionally passed be passed in.
> HTableInterfaces are retrieved from the HConnection. By default the
> HConnection's ExecutorService is used, but optionally that can be
> overridden for each HTable.
>
> In 0.98/trunk:
>
> 1. HTablePool will be removed. It is not longer needed.
> 2. All constructors in HTable will be removed and changed to be protected.
> All code use HTableInterface only.
> 3. HConnectionManager.getConnection() will be removed.
> 3. All HConnection caching (deleteConnection, etc,etc) will be removed, as
> it is no longer needed.
>
>
> The new flow of setting up a client would look like this:
>
> ----- Snip -----
> // connection to the cluster
> HConnection conn = HConnectionManager.createConnection(conf);
> ...
> // When the cluster connection is established get an HTableInterface for
> each operation or thread.
> // HConnection.getTable(...) is lightweight. The table is really just a
> convenient place to call table method and for a temporary batch cache.
> // It is in fact less overhead than HTablePool had when retrieving a
> cached HTable.
> // The HTableInterface returned is not thread safe as before.
> // It's fine to get 1000's of these.
> // Don't cache the longer than the lifetime of the HConnection
> HTableInterface table = conn.getTable("MyTable");
> ...
> // just flushes outstanding commit, no futher cleanup needed, can be
> omitted.
> // HConnection holds no references to the returned HTable objects, they
> can be GC'd as soon as they leave scope.
> table.close();
> ...
> conn.close(); // done with the cluster, release resources
> ----- Snip -----
>
> The HConnection will maintain and share its own ThreadPool for all batch
> operations executed by the HTables.
> This can overridden per HConnection and/or per individual HTable object.
>
> I will commit the new API to all branches early next week.
>
> Questions? Comments? Concerns? Praise?
>
> -- Lars