You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Adam Fuchs <af...@apache.org> on 2016/08/15 17:49:06 UTC
bulk load architecture
I've been looking through the bulk load code lately related to some
performance issues a customer of ours is experiencing, and I'm perplexed by
a couple of things. Between o.a.a.master.tableOps.LoadFiles and
o.a.a.server.client.BulkImporter we have 4 thread pools that are used in
bulk load. It seems like only the master thread pool gets any parallelism
because we always send one file at a time to the tservers (LoadFiles:154).
Are the three thread pools in the tserver vestigial? Did we used to send
bigger batches to the tservers and find that one at a time was more optimal?
Seems like we could greatly simplify the tserver portion of the bulk load.
Can anybody think of why that might not be a good idea?
Also, has anybody optimized the pool sizes for multiple concurrent large
bulk loads, and do you have suggestions on what settings to use (i.e.
master.fate.threadpool.size and master.bulk.threadpool.size)?
Thanks,
Adam
Re: bulk load architecture
Posted by Adam Fuchs <af...@apache.org>.
Thanks, Chris. I would still appreciate any anecdotes people have for
setting these thread pools.
I did a bit more log analysis on bulk import times. With the default of 5
threads we're seeing an average somewhere around 50-100ms per file per
thread in the master bulk import thread pool. This means that a bulk import
of 1000 files (e.g. 1 file for each of 1000 tablets) takes about 10-20
seconds. We're targeting something like 1 bulk import per second for some
of our instances, most of which are much smaller than 1000 files, and we're
going to need better performance in that scenario. Our plan is to run with
200 threads for master.bulk.threadpool.size for a while on one of our
larger clusters and see how well that works. Thoughts?
Adam
On Thu, Aug 18, 2016 at 10:39 AM, Christopher <ct...@apache.org> wrote:
> Bumping this thread up, because I'm also curious if anybody has any
> thoughts on Adam's questions.
>
> On Mon, Aug 15, 2016 at 1:49 PM Adam Fuchs <af...@apache.org> wrote:
>
> > I've been looking through the bulk load code lately related to some
> > performance issues a customer of ours is experiencing, and I'm perplexed
> by
> > a couple of things. Between o.a.a.master.tableOps.LoadFiles and
> > o.a.a.server.client.BulkImporter we have 4 thread pools that are used in
> > bulk load. It seems like only the master thread pool gets any parallelism
> > because we always send one file at a time to the tservers
> (LoadFiles:154).
> > Are the three thread pools in the tserver vestigial? Did we used to send
> > bigger batches to the tservers and find that one at a time was more
> > optimal?
> >
> > Seems like we could greatly simplify the tserver portion of the bulk
> load.
> > Can anybody think of why that might not be a good idea?
> >
> > Also, has anybody optimized the pool sizes for multiple concurrent large
> > bulk loads, and do you have suggestions on what settings to use (i.e.
> > master.fate.threadpool.size and master.bulk.threadpool.size)?
> >
> > Thanks,
> > Adam
> >
>
Re: bulk load architecture
Posted by Christopher <ct...@apache.org>.
Bumping this thread up, because I'm also curious if anybody has any
thoughts on Adam's questions.
On Mon, Aug 15, 2016 at 1:49 PM Adam Fuchs <af...@apache.org> wrote:
> I've been looking through the bulk load code lately related to some
> performance issues a customer of ours is experiencing, and I'm perplexed by
> a couple of things. Between o.a.a.master.tableOps.LoadFiles and
> o.a.a.server.client.BulkImporter we have 4 thread pools that are used in
> bulk load. It seems like only the master thread pool gets any parallelism
> because we always send one file at a time to the tservers (LoadFiles:154).
> Are the three thread pools in the tserver vestigial? Did we used to send
> bigger batches to the tservers and find that one at a time was more
> optimal?
>
> Seems like we could greatly simplify the tserver portion of the bulk load.
> Can anybody think of why that might not be a good idea?
>
> Also, has anybody optimized the pool sizes for multiple concurrent large
> bulk loads, and do you have suggestions on what settings to use (i.e.
> master.fate.threadpool.size and master.bulk.threadpool.size)?
>
> Thanks,
> Adam
>