You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Charles Thayer <th...@b2si.com> on 2011/08/26 08:12:47 UTC
HTable's thread pool question
Would it make sense to have all HTables share a single thread pool, and
possibly use org.apache.hadoop.hbase.util.ShardedBlockingQueue instead
of SynchronousQueue (trunk) and instead of LinkedBlockingQueue? This
seems more in-sync with the HConnectionManager model. I understand
there's a HTablePool, and this would be a different approach to the same
problem.
I ran into this problem where our app was had > 10,000 threads. We were
creating 100 objects (used by many threads) that each have 9 HTables.
We have ~70 region servers (NrHRS), and we're using an early 0.90
client/server with some random patches. Given the code below, I
realized we were going to allocate up to 100 thread pools of 9*70
threads each. I changed the corePoolSize to 1, and later found that
this issue is already fixed in every version after the one we're using
:) and in trunk it's been switched to a SynchronousQueue.
Thanks, sorry if this has been discussed,
/charles thayer
(aka cgthayer@fb.com)
hbase-0.90-local::
----------------
192 int nrThreads = conf.getInt("hbase.htable.threads.max",
getCurrentNrHRS());
193 if (nrThreads == 0) {
194 nrThreads = 1; // is there a better default?
195 }
196
197 // Unfortunately Executors.newCachedThreadPool does not
allow us to
198 // set the maximum size of the pool, so we have to do it
ourselves.
199 // Must also set set corethreadpool size as with a
LinkedBlockingQueue,
200 // a new thread will not be started until the queue is full
201 this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
202 60, TimeUnit.SECONDS,
203 new LinkedBlockingQueue<Runnable>(),
204 new DaemonThreadFactory());
205 ((ThreadPoolExecutor)this.pool).allowCoreThreadTimeOut(true);
hbase-0.90.1::
-------------
182 int nrThreads = conf.getInt("hbase.htable.threads.max",
getCurrentNrHRS());
183 if (nrThreads == 0) {
184 nrThreads = 1; // is there a better default?
185 }
186
187 // Unfortunately Executors.newCachedThreadPool does not
allow us to
188 // set the maximum size of the pool, so we have to do it
ourselves.
189 this.pool = new ThreadPoolExecutor(0, nrThreads,
190 60, TimeUnit.SECONDS,
191 new LinkedBlockingQueue<Runnable>(),
192 new DaemonThreadFactory());
193 }
Current Trunk::
-------------
1024074 rawson
1094741 jdcryans int maxThreads =
conf.getInt("hbase.htable.threads.max", Integer.MAX_VALUE);
1094741 jdcryans if (maxThreads == 0) {
1094741 jdcryans maxThreads = 1; // is there a better default?
909720 stack }
909235 rawson
1094741 jdcryans // Using the "direct handoff" approach, new
threads will only be created
1094741 jdcryans // if it is necessary and will grow unbounded.
This could be bad but in HCM
1094741 jdcryans // we only create as many Runnables as there are
region servers. It means
1094741 jdcryans // it also scales when new region servers are added.
1094741 jdcryans this.pool = new ThreadPoolExecutor(1, maxThreads,
909235 rawson 60, TimeUnit.SECONDS,
1094741 jdcryans new SynchronousQueue<Runnable>(),
909235 rawson new DaemonThreadFactory());
1074331 garyh
((ThreadPoolExecutor)this.pool).allowCoreThreadTimeOut(true);
1098901 stack this.closed = false;
685009 jimk }
685009 jimk
Re: HTable's thread pool question
Posted by Ted Yu <yu...@gmail.com>.
This sounds like a nice addition.
Would you mind contributing ShardedBlockQueue to open source ?
Thanks
On Fri, Aug 26, 2011 at 9:05 AM, Charles Thayer <cg...@fb.com> wrote:
> Woops, my bad, ShardedBlockQueue is a local update and hasn't made it to a
> jira yet. It's just a queue split among N sub-queues, to relieve lock
> contention.
> /c
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, August 26, 2011 7:58 AM
> To: dev@hbase.apache.org
> Cc: jdcryans@gmail.com; vashishtha.h@gmail.com
> Subject: Re: HTable's thread pool question
>
> I cannot find ShardedBlockingQueue in hbase code base.
> Can you point me to where the class resides ?
>
> Thanks
>
> On Thu, Aug 25, 2011 at 11:12 PM, Charles Thayer <th...@b2si.com> wrote:
>
> > Would it make sense to have all HTables share a single thread pool,
> > and possibly use org.apache.hadoop.hbase.util.**ShardedBlockingQueue
> > instead of SynchronousQueue (trunk) and instead of
> > LinkedBlockingQueue? This seems more in-sync with the
> > HConnectionManager model. I understand there's a HTablePool, and this
> would be a different approach to the same problem.
> >
> > I ran into this problem where our app was had > 10,000 threads. We
> > were creating 100 objects (used by many threads) that each have 9
> > HTables. We have ~70 region servers (NrHRS), and we're using an early
> > 0.90 client/server with some random patches. Given the code below, I
> > realized we were going to allocate up to 100 thread pools of 9*70
> > threads each. I changed the corePoolSize to 1, and later found that
> > this issue is already fixed in every version after the one we're using
> > :) and in trunk it's been switched to a SynchronousQueue.
> >
> > Thanks, sorry if this has been discussed, /charles thayer (aka
> > cgthayer@fb.com)
> >
> >
> > hbase-0.90-local::
> > ----------------
> > 192 int nrThreads = conf.getInt("hbase.htable.**threads.max",
> > getCurrentNrHRS());
> > 193 if (nrThreads == 0) {
> > 194 nrThreads = 1; // is there a better default?
> > 195 }
> > 196
> > 197 // Unfortunately Executors.newCachedThreadPool does not allow
> us
> > to
> > 198 // set the maximum size of the pool, so we have to do it
> > ourselves.
> > 199 // Must also set set corethreadpool size as with a
> > LinkedBlockingQueue,
> > 200 // a new thread will not be started until the queue is full
> > 201 this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
> > 202 60, TimeUnit.SECONDS,
> > 203 new LinkedBlockingQueue<Runnable>(**),
> > 204 new DaemonThreadFactory());
> > 205 ((ThreadPoolExecutor)this.**pool).allowCoreThreadTimeOut(**
> > true);
> >
> > hbase-0.90.1::
> > -------------
> > 182 int nrThreads = conf.getInt("hbase.htable.**threads.max",
> > getCurrentNrHRS());
> > 183 if (nrThreads == 0) {
> > 184 nrThreads = 1; // is there a better default?
> > 185 }
> > 186
> > 187 // Unfortunately Executors.newCachedThreadPool does not allow
> us
> > to
> > 188 // set the maximum size of the pool, so we have to do it
> > ourselves.
> > 189 this.pool = new ThreadPoolExecutor(0, nrThreads,
> > 190 60, TimeUnit.SECONDS,
> > 191 new LinkedBlockingQueue<Runnable>(**),
> > 192 new DaemonThreadFactory());
> > 193 }
> >
> > Current Trunk::
> > -------------
> > 1024074 rawson
> > 1094741 jdcryans int maxThreads =
> conf.getInt("hbase.htable.**threads.max",
> > Integer.MAX_VALUE);
> > 1094741 jdcryans if (maxThreads == 0) {
> > 1094741 jdcryans maxThreads = 1; // is there a better default?
> > 909720 stack }
> > 909235 rawson
> > 1094741 jdcryans // Using the "direct handoff" approach, new
> threads
> > will only be created
> > 1094741 jdcryans // if it is necessary and will grow unbounded.
> This
> > could be bad but in HCM
> > 1094741 jdcryans // we only create as many Runnables as there are
> > region servers. It means
> > 1094741 jdcryans // it also scales when new region servers are
> added.
> > 1094741 jdcryans this.pool = new ThreadPoolExecutor(1, maxThreads,
> > 909235 rawson 60, TimeUnit.SECONDS,
> > 1094741 jdcryans new SynchronousQueue<Runnable>(),
> > 909235 rawson new DaemonThreadFactory());
> > 1074331 garyh ((ThreadPoolExecutor)this.**
> > pool).allowCoreThreadTimeOut(**true);
> > 1098901 stack this.closed = false;
> > 685009 jimk }
> > 685009 jimk
> >
> >
>
RE: HTable's thread pool question
Posted by Charles Thayer <cg...@fb.com>.
Woops, my bad, ShardedBlockQueue is a local update and hasn't made it to a jira yet. It's just a queue split among N sub-queues, to relieve lock contention.
/c
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, August 26, 2011 7:58 AM
To: dev@hbase.apache.org
Cc: jdcryans@gmail.com; vashishtha.h@gmail.com
Subject: Re: HTable's thread pool question
I cannot find ShardedBlockingQueue in hbase code base.
Can you point me to where the class resides ?
Thanks
On Thu, Aug 25, 2011 at 11:12 PM, Charles Thayer <th...@b2si.com> wrote:
> Would it make sense to have all HTables share a single thread pool,
> and possibly use org.apache.hadoop.hbase.util.**ShardedBlockingQueue
> instead of SynchronousQueue (trunk) and instead of
> LinkedBlockingQueue? This seems more in-sync with the
> HConnectionManager model. I understand there's a HTablePool, and this would be a different approach to the same problem.
>
> I ran into this problem where our app was had > 10,000 threads. We
> were creating 100 objects (used by many threads) that each have 9
> HTables. We have ~70 region servers (NrHRS), and we're using an early
> 0.90 client/server with some random patches. Given the code below, I
> realized we were going to allocate up to 100 thread pools of 9*70
> threads each. I changed the corePoolSize to 1, and later found that
> this issue is already fixed in every version after the one we're using
> :) and in trunk it's been switched to a SynchronousQueue.
>
> Thanks, sorry if this has been discussed, /charles thayer (aka
> cgthayer@fb.com)
>
>
> hbase-0.90-local::
> ----------------
> 192 int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
> 193 if (nrThreads == 0) {
> 194 nrThreads = 1; // is there a better default?
> 195 }
> 196
> 197 // Unfortunately Executors.newCachedThreadPool does not allow us
> to
> 198 // set the maximum size of the pool, so we have to do it
> ourselves.
> 199 // Must also set set corethreadpool size as with a
> LinkedBlockingQueue,
> 200 // a new thread will not be started until the queue is full
> 201 this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
> 202 60, TimeUnit.SECONDS,
> 203 new LinkedBlockingQueue<Runnable>(**),
> 204 new DaemonThreadFactory());
> 205 ((ThreadPoolExecutor)this.**pool).allowCoreThreadTimeOut(**
> true);
>
> hbase-0.90.1::
> -------------
> 182 int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
> 183 if (nrThreads == 0) {
> 184 nrThreads = 1; // is there a better default?
> 185 }
> 186
> 187 // Unfortunately Executors.newCachedThreadPool does not allow us
> to
> 188 // set the maximum size of the pool, so we have to do it
> ourselves.
> 189 this.pool = new ThreadPoolExecutor(0, nrThreads,
> 190 60, TimeUnit.SECONDS,
> 191 new LinkedBlockingQueue<Runnable>(**),
> 192 new DaemonThreadFactory());
> 193 }
>
> Current Trunk::
> -------------
> 1024074 rawson
> 1094741 jdcryans int maxThreads = conf.getInt("hbase.htable.**threads.max",
> Integer.MAX_VALUE);
> 1094741 jdcryans if (maxThreads == 0) {
> 1094741 jdcryans maxThreads = 1; // is there a better default?
> 909720 stack }
> 909235 rawson
> 1094741 jdcryans // Using the "direct handoff" approach, new threads
> will only be created
> 1094741 jdcryans // if it is necessary and will grow unbounded. This
> could be bad but in HCM
> 1094741 jdcryans // we only create as many Runnables as there are
> region servers. It means
> 1094741 jdcryans // it also scales when new region servers are added.
> 1094741 jdcryans this.pool = new ThreadPoolExecutor(1, maxThreads,
> 909235 rawson 60, TimeUnit.SECONDS,
> 1094741 jdcryans new SynchronousQueue<Runnable>(),
> 909235 rawson new DaemonThreadFactory());
> 1074331 garyh ((ThreadPoolExecutor)this.**
> pool).allowCoreThreadTimeOut(**true);
> 1098901 stack this.closed = false;
> 685009 jimk }
> 685009 jimk
>
>
Re: HTable's thread pool question
Posted by Ted Yu <yu...@gmail.com>.
I cannot find ShardedBlockingQueue in hbase code base.
Can you point me to where the class resides ?
Thanks
On Thu, Aug 25, 2011 at 11:12 PM, Charles Thayer <th...@b2si.com> wrote:
> Would it make sense to have all HTables share a single thread pool, and
> possibly use org.apache.hadoop.hbase.util.**ShardedBlockingQueue instead
> of SynchronousQueue (trunk) and instead of LinkedBlockingQueue? This seems
> more in-sync with the HConnectionManager model. I understand there's a
> HTablePool, and this would be a different approach to the same problem.
>
> I ran into this problem where our app was had > 10,000 threads. We were
> creating 100 objects (used by many threads) that each have 9 HTables. We
> have ~70 region servers (NrHRS), and we're using an early 0.90 client/server
> with some random patches. Given the code below, I realized we were going to
> allocate up to 100 thread pools of 9*70 threads each. I changed the
> corePoolSize to 1, and later found that this issue is already fixed in every
> version after the one we're using :) and in trunk it's been switched to a
> SynchronousQueue.
>
> Thanks, sorry if this has been discussed,
> /charles thayer
> (aka cgthayer@fb.com)
>
>
> hbase-0.90-local::
> ----------------
> 192 int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
> 193 if (nrThreads == 0) {
> 194 nrThreads = 1; // is there a better default?
> 195 }
> 196
> 197 // Unfortunately Executors.newCachedThreadPool does not allow us
> to
> 198 // set the maximum size of the pool, so we have to do it
> ourselves.
> 199 // Must also set set corethreadpool size as with a
> LinkedBlockingQueue,
> 200 // a new thread will not be started until the queue is full
> 201 this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
> 202 60, TimeUnit.SECONDS,
> 203 new LinkedBlockingQueue<Runnable>(**),
> 204 new DaemonThreadFactory());
> 205 ((ThreadPoolExecutor)this.**pool).allowCoreThreadTimeOut(**
> true);
>
> hbase-0.90.1::
> -------------
> 182 int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
> 183 if (nrThreads == 0) {
> 184 nrThreads = 1; // is there a better default?
> 185 }
> 186
> 187 // Unfortunately Executors.newCachedThreadPool does not allow us
> to
> 188 // set the maximum size of the pool, so we have to do it
> ourselves.
> 189 this.pool = new ThreadPoolExecutor(0, nrThreads,
> 190 60, TimeUnit.SECONDS,
> 191 new LinkedBlockingQueue<Runnable>(**),
> 192 new DaemonThreadFactory());
> 193 }
>
> Current Trunk::
> -------------
> 1024074 rawson
> 1094741 jdcryans int maxThreads = conf.getInt("hbase.htable.**threads.max",
> Integer.MAX_VALUE);
> 1094741 jdcryans if (maxThreads == 0) {
> 1094741 jdcryans maxThreads = 1; // is there a better default?
> 909720 stack }
> 909235 rawson
> 1094741 jdcryans // Using the "direct handoff" approach, new threads
> will only be created
> 1094741 jdcryans // if it is necessary and will grow unbounded. This
> could be bad but in HCM
> 1094741 jdcryans // we only create as many Runnables as there are
> region servers. It means
> 1094741 jdcryans // it also scales when new region servers are added.
> 1094741 jdcryans this.pool = new ThreadPoolExecutor(1, maxThreads,
> 909235 rawson 60, TimeUnit.SECONDS,
> 1094741 jdcryans new SynchronousQueue<Runnable>(),
> 909235 rawson new DaemonThreadFactory());
> 1074331 garyh ((ThreadPoolExecutor)this.**
> pool).allowCoreThreadTimeOut(**true);
> 1098901 stack this.closed = false;
> 685009 jimk }
> 685009 jimk
>
>