You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Charles Thayer <th...@b2si.com> on 2011/08/26 08:12:47 UTC

HTable's thread pool question

Would it make sense to have all HTables share a single thread pool, and 
possibly use org.apache.hadoop.hbase.util.ShardedBlockingQueue instead 
of SynchronousQueue (trunk) and instead of LinkedBlockingQueue?  This 
seems more in-sync with the HConnectionManager model.  I understand 
there's a HTablePool, and this would be a different approach to the same 
problem.

I ran into this problem where our app was had > 10,000 threads.  We were 
creating 100 objects (used by many threads) that each have 9 HTables.  
We have ~70 region servers (NrHRS), and we're using an early 0.90 
client/server with some random patches.  Given the code below, I 
realized we were going to allocate up to 100 thread pools of 9*70 
threads each.  I changed the corePoolSize to 1, and later found that 
this issue is already fixed in every version after the one we're using  
:)  and in trunk it's been switched to a SynchronousQueue.

Thanks, sorry if this has been discussed,
/charles thayer
(aka cgthayer@fb.com)


hbase-0.90-local::
----------------
     192     int nrThreads = conf.getInt("hbase.htable.threads.max", 
getCurrentNrHRS());
     193     if (nrThreads == 0) {
     194       nrThreads = 1; // is there a better default?
     195     }
     196
     197     // Unfortunately Executors.newCachedThreadPool does not 
allow us to
     198     // set the maximum size of the pool, so we have to do it 
ourselves.
     199     // Must also set set corethreadpool size as with a 
LinkedBlockingQueue,
     200     // a new thread will not be started until the queue is full
     201     this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
     202         60, TimeUnit.SECONDS,
     203         new LinkedBlockingQueue<Runnable>(),
     204         new DaemonThreadFactory());
     205     ((ThreadPoolExecutor)this.pool).allowCoreThreadTimeOut(true);

hbase-0.90.1::
-------------
     182     int nrThreads = conf.getInt("hbase.htable.threads.max", 
getCurrentNrHRS());
     183     if (nrThreads == 0) {
     184       nrThreads = 1; // is there a better default?
     185     }
     186
     187     // Unfortunately Executors.newCachedThreadPool does not 
allow us to
     188     // set the maximum size of the pool, so we have to do it 
ourselves.
     189     this.pool = new ThreadPoolExecutor(0, nrThreads,
     190         60, TimeUnit.SECONDS,
     191         new LinkedBlockingQueue<Runnable>(),
     192         new DaemonThreadFactory());
     193   }

Current Trunk::
-------------
1024074     rawson
1094741   jdcryans     int maxThreads = 
conf.getInt("hbase.htable.threads.max", Integer.MAX_VALUE);
1094741   jdcryans     if (maxThreads == 0) {
1094741   jdcryans       maxThreads = 1; // is there a better default?
909720      stack     }
909235     rawson
1094741   jdcryans     // Using the "direct handoff" approach, new 
threads will only be created
1094741   jdcryans     // if it is necessary and will grow unbounded. 
This could be bad but in HCM
1094741   jdcryans     // we only create as many Runnables as there are 
region servers. It means
1094741   jdcryans     // it also scales when new region servers are added.
1094741   jdcryans     this.pool = new ThreadPoolExecutor(1, maxThreads,
909235     rawson         60, TimeUnit.SECONDS,
1094741   jdcryans         new SynchronousQueue<Runnable>(),
909235     rawson         new DaemonThreadFactory());
1074331      garyh     
((ThreadPoolExecutor)this.pool).allowCoreThreadTimeOut(true);
1098901      stack     this.closed = false;
685009       jimk   }
685009       jimk


Re: HTable's thread pool question

Posted by Ted Yu <yu...@gmail.com>.
This sounds like a nice addition.
Would you mind contributing ShardedBlockQueue to open source ?

Thanks

On Fri, Aug 26, 2011 at 9:05 AM, Charles Thayer <cg...@fb.com> wrote:

> Woops, my bad, ShardedBlockQueue is a local update and hasn't made it to a
> jira yet.  It's just a queue split among N sub-queues, to relieve lock
> contention.
> /c
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, August 26, 2011 7:58 AM
> To: dev@hbase.apache.org
> Cc: jdcryans@gmail.com; vashishtha.h@gmail.com
> Subject: Re: HTable's thread pool question
>
> I cannot find ShardedBlockingQueue in hbase code base.
> Can you point me to where the class resides ?
>
> Thanks
>
> On Thu, Aug 25, 2011 at 11:12 PM, Charles Thayer <th...@b2si.com> wrote:
>
> > Would it make sense to have all HTables share a single thread pool,
> > and possibly use org.apache.hadoop.hbase.util.**ShardedBlockingQueue
> > instead of SynchronousQueue (trunk) and instead of
> > LinkedBlockingQueue?  This seems more in-sync with the
> > HConnectionManager model.  I understand there's a HTablePool, and this
> would be a different approach to the same problem.
> >
> > I ran into this problem where our app was had > 10,000 threads.  We
> > were creating 100 objects (used by many threads) that each have 9
> > HTables.  We have ~70 region servers (NrHRS), and we're using an early
> > 0.90 client/server with some random patches.  Given the code below, I
> > realized we were going to allocate up to 100 thread pools of 9*70
> > threads each.  I changed the corePoolSize to 1, and later found that
> > this issue is already fixed in every version after the one we're using
> > :)  and in trunk it's been switched to a SynchronousQueue.
> >
> > Thanks, sorry if this has been discussed, /charles thayer (aka
> > cgthayer@fb.com)
> >
> >
> > hbase-0.90-local::
> > ----------------
> >    192     int nrThreads = conf.getInt("hbase.htable.**threads.max",
> > getCurrentNrHRS());
> >    193     if (nrThreads == 0) {
> >    194       nrThreads = 1; // is there a better default?
> >    195     }
> >    196
> >    197     // Unfortunately Executors.newCachedThreadPool does not allow
> us
> > to
> >    198     // set the maximum size of the pool, so we have to do it
> > ourselves.
> >    199     // Must also set set corethreadpool size as with a
> > LinkedBlockingQueue,
> >    200     // a new thread will not be started until the queue is full
> >    201     this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
> >    202         60, TimeUnit.SECONDS,
> >    203         new LinkedBlockingQueue<Runnable>(**),
> >    204         new DaemonThreadFactory());
> >    205     ((ThreadPoolExecutor)this.**pool).allowCoreThreadTimeOut(**
> > true);
> >
> > hbase-0.90.1::
> > -------------
> >    182     int nrThreads = conf.getInt("hbase.htable.**threads.max",
> > getCurrentNrHRS());
> >    183     if (nrThreads == 0) {
> >    184       nrThreads = 1; // is there a better default?
> >    185     }
> >    186
> >    187     // Unfortunately Executors.newCachedThreadPool does not allow
> us
> > to
> >    188     // set the maximum size of the pool, so we have to do it
> > ourselves.
> >    189     this.pool = new ThreadPoolExecutor(0, nrThreads,
> >    190         60, TimeUnit.SECONDS,
> >    191         new LinkedBlockingQueue<Runnable>(**),
> >    192         new DaemonThreadFactory());
> >    193   }
> >
> > Current Trunk::
> > -------------
> > 1024074     rawson
> > 1094741   jdcryans     int maxThreads =
> conf.getInt("hbase.htable.**threads.max",
> > Integer.MAX_VALUE);
> > 1094741   jdcryans     if (maxThreads == 0) {
> > 1094741   jdcryans       maxThreads = 1; // is there a better default?
> > 909720      stack     }
> > 909235     rawson
> > 1094741   jdcryans     // Using the "direct handoff" approach, new
> threads
> > will only be created
> > 1094741   jdcryans     // if it is necessary and will grow unbounded.
> This
> > could be bad but in HCM
> > 1094741   jdcryans     // we only create as many Runnables as there are
> > region servers. It means
> > 1094741   jdcryans     // it also scales when new region servers are
> added.
> > 1094741   jdcryans     this.pool = new ThreadPoolExecutor(1, maxThreads,
> > 909235     rawson         60, TimeUnit.SECONDS,
> > 1094741   jdcryans         new SynchronousQueue<Runnable>(),
> > 909235     rawson         new DaemonThreadFactory());
> > 1074331      garyh     ((ThreadPoolExecutor)this.**
> > pool).allowCoreThreadTimeOut(**true);
> > 1098901      stack     this.closed = false;
> > 685009       jimk   }
> > 685009       jimk
> >
> >
>

RE: HTable's thread pool question

Posted by Charles Thayer <cg...@fb.com>.
Woops, my bad, ShardedBlockQueue is a local update and hasn't made it to a jira yet.  It's just a queue split among N sub-queues, to relieve lock contention.
/c 

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Friday, August 26, 2011 7:58 AM
To: dev@hbase.apache.org
Cc: jdcryans@gmail.com; vashishtha.h@gmail.com
Subject: Re: HTable's thread pool question

I cannot find ShardedBlockingQueue in hbase code base.
Can you point me to where the class resides ?

Thanks

On Thu, Aug 25, 2011 at 11:12 PM, Charles Thayer <th...@b2si.com> wrote:

> Would it make sense to have all HTables share a single thread pool, 
> and possibly use org.apache.hadoop.hbase.util.**ShardedBlockingQueue 
> instead of SynchronousQueue (trunk) and instead of 
> LinkedBlockingQueue?  This seems more in-sync with the 
> HConnectionManager model.  I understand there's a HTablePool, and this would be a different approach to the same problem.
>
> I ran into this problem where our app was had > 10,000 threads.  We 
> were creating 100 objects (used by many threads) that each have 9 
> HTables.  We have ~70 region servers (NrHRS), and we're using an early 
> 0.90 client/server with some random patches.  Given the code below, I 
> realized we were going to allocate up to 100 thread pools of 9*70 
> threads each.  I changed the corePoolSize to 1, and later found that 
> this issue is already fixed in every version after the one we're using  
> :)  and in trunk it's been switched to a SynchronousQueue.
>
> Thanks, sorry if this has been discussed, /charles thayer (aka 
> cgthayer@fb.com)
>
>
> hbase-0.90-local::
> ----------------
>    192     int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
>    193     if (nrThreads == 0) {
>    194       nrThreads = 1; // is there a better default?
>    195     }
>    196
>    197     // Unfortunately Executors.newCachedThreadPool does not allow us
> to
>    198     // set the maximum size of the pool, so we have to do it
> ourselves.
>    199     // Must also set set corethreadpool size as with a
> LinkedBlockingQueue,
>    200     // a new thread will not be started until the queue is full
>    201     this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
>    202         60, TimeUnit.SECONDS,
>    203         new LinkedBlockingQueue<Runnable>(**),
>    204         new DaemonThreadFactory());
>    205     ((ThreadPoolExecutor)this.**pool).allowCoreThreadTimeOut(**
> true);
>
> hbase-0.90.1::
> -------------
>    182     int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
>    183     if (nrThreads == 0) {
>    184       nrThreads = 1; // is there a better default?
>    185     }
>    186
>    187     // Unfortunately Executors.newCachedThreadPool does not allow us
> to
>    188     // set the maximum size of the pool, so we have to do it
> ourselves.
>    189     this.pool = new ThreadPoolExecutor(0, nrThreads,
>    190         60, TimeUnit.SECONDS,
>    191         new LinkedBlockingQueue<Runnable>(**),
>    192         new DaemonThreadFactory());
>    193   }
>
> Current Trunk::
> -------------
> 1024074     rawson
> 1094741   jdcryans     int maxThreads = conf.getInt("hbase.htable.**threads.max",
> Integer.MAX_VALUE);
> 1094741   jdcryans     if (maxThreads == 0) {
> 1094741   jdcryans       maxThreads = 1; // is there a better default?
> 909720      stack     }
> 909235     rawson
> 1094741   jdcryans     // Using the "direct handoff" approach, new threads
> will only be created
> 1094741   jdcryans     // if it is necessary and will grow unbounded. This
> could be bad but in HCM
> 1094741   jdcryans     // we only create as many Runnables as there are
> region servers. It means
> 1094741   jdcryans     // it also scales when new region servers are added.
> 1094741   jdcryans     this.pool = new ThreadPoolExecutor(1, maxThreads,
> 909235     rawson         60, TimeUnit.SECONDS,
> 1094741   jdcryans         new SynchronousQueue<Runnable>(),
> 909235     rawson         new DaemonThreadFactory());
> 1074331      garyh     ((ThreadPoolExecutor)this.**
> pool).allowCoreThreadTimeOut(**true);
> 1098901      stack     this.closed = false;
> 685009       jimk   }
> 685009       jimk
>
>

Re: HTable's thread pool question

Posted by Ted Yu <yu...@gmail.com>.
I cannot find ShardedBlockingQueue in hbase code base.
Can you point me to where the class resides ?

Thanks

On Thu, Aug 25, 2011 at 11:12 PM, Charles Thayer <th...@b2si.com> wrote:

> Would it make sense to have all HTables share a single thread pool, and
> possibly use org.apache.hadoop.hbase.util.**ShardedBlockingQueue instead
> of SynchronousQueue (trunk) and instead of LinkedBlockingQueue?  This seems
> more in-sync with the HConnectionManager model.  I understand there's a
> HTablePool, and this would be a different approach to the same problem.
>
> I ran into this problem where our app was had > 10,000 threads.  We were
> creating 100 objects (used by many threads) that each have 9 HTables.  We
> have ~70 region servers (NrHRS), and we're using an early 0.90 client/server
> with some random patches.  Given the code below, I realized we were going to
> allocate up to 100 thread pools of 9*70 threads each.  I changed the
> corePoolSize to 1, and later found that this issue is already fixed in every
> version after the one we're using  :)  and in trunk it's been switched to a
> SynchronousQueue.
>
> Thanks, sorry if this has been discussed,
> /charles thayer
> (aka cgthayer@fb.com)
>
>
> hbase-0.90-local::
> ----------------
>    192     int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
>    193     if (nrThreads == 0) {
>    194       nrThreads = 1; // is there a better default?
>    195     }
>    196
>    197     // Unfortunately Executors.newCachedThreadPool does not allow us
> to
>    198     // set the maximum size of the pool, so we have to do it
> ourselves.
>    199     // Must also set set corethreadpool size as with a
> LinkedBlockingQueue,
>    200     // a new thread will not be started until the queue is full
>    201     this.pool = new ThreadPoolExecutor(nrThreads, nrThreads,
>    202         60, TimeUnit.SECONDS,
>    203         new LinkedBlockingQueue<Runnable>(**),
>    204         new DaemonThreadFactory());
>    205     ((ThreadPoolExecutor)this.**pool).allowCoreThreadTimeOut(**
> true);
>
> hbase-0.90.1::
> -------------
>    182     int nrThreads = conf.getInt("hbase.htable.**threads.max",
> getCurrentNrHRS());
>    183     if (nrThreads == 0) {
>    184       nrThreads = 1; // is there a better default?
>    185     }
>    186
>    187     // Unfortunately Executors.newCachedThreadPool does not allow us
> to
>    188     // set the maximum size of the pool, so we have to do it
> ourselves.
>    189     this.pool = new ThreadPoolExecutor(0, nrThreads,
>    190         60, TimeUnit.SECONDS,
>    191         new LinkedBlockingQueue<Runnable>(**),
>    192         new DaemonThreadFactory());
>    193   }
>
> Current Trunk::
> -------------
> 1024074     rawson
> 1094741   jdcryans     int maxThreads = conf.getInt("hbase.htable.**threads.max",
> Integer.MAX_VALUE);
> 1094741   jdcryans     if (maxThreads == 0) {
> 1094741   jdcryans       maxThreads = 1; // is there a better default?
> 909720      stack     }
> 909235     rawson
> 1094741   jdcryans     // Using the "direct handoff" approach, new threads
> will only be created
> 1094741   jdcryans     // if it is necessary and will grow unbounded. This
> could be bad but in HCM
> 1094741   jdcryans     // we only create as many Runnables as there are
> region servers. It means
> 1094741   jdcryans     // it also scales when new region servers are added.
> 1094741   jdcryans     this.pool = new ThreadPoolExecutor(1, maxThreads,
> 909235     rawson         60, TimeUnit.SECONDS,
> 1094741   jdcryans         new SynchronousQueue<Runnable>(),
> 909235     rawson         new DaemonThreadFactory());
> 1074331      garyh     ((ThreadPoolExecutor)this.**
> pool).allowCoreThreadTimeOut(**true);
> 1098901      stack     this.closed = false;
> 685009       jimk   }
> 685009       jimk
>
>