You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jianshi Huang <ji...@gmail.com> on 2015/03/02 10:42:56 UTC
Need Help: RegionTooBusyException: Above memstore limit
Hi,
I'm constantly facing "RegionTooBusyException: Above memstore limit" errors
in one region server when writing data to HBase.
I checked the region server log, and I've seen a lot of warnings during the
data writes:
WARN wal.fshlog couldn't find oldest seqNum for the region we're about to
flush, ...
Then HBase seem to flush the data and added it as a HStore file.
I also get a few warnings in client.ShortCircuitCache, says "could not load
... due to InvalidToken exceptions.
Anyone can give me hint what went wrong?
My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the release
is a little bit old.
Thanks,
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jianshi Huang <ji...@gmail.com>.
The error disappeared after changing write buffer from 20MB to 2MB. Thanks
for the help!
Jianshi
On Wed, Mar 4, 2015 at 12:12 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> It depends on how you manage your connection, your table and your puts. If
> it works for you with reducing the batch buffer size, then just keep it the
> way it is...
>
> JM
>
> 2015-03-03 11:10 GMT-05:00 Jianshi Huang <ji...@gmail.com>:
>
> > Yes, looks like reducing the batch buffer size works (still validating).
> >
> > But why setAutoFlush(false) is harmful here? I just want maximum write
> > speed.
> >
> > Jianshi
> >
> > On Tue, Mar 3, 2015 at 10:54 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > Let HBase manage the flushes for you. Remove
> > edgeTable.setAutoFlush(false)
> > > and maybe reduce your batch size.
> > >
> > > I don't think that increasing the memstore is the good way to go. Sound
> > > more like a plaster on the issue than a good fix (for me).
> > >
> > > JM
> > >
> > > 2015-03-03 9:43 GMT-05:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > Default value for hbase.regionserver.global.memstore.size is 0.4
> > > >
> > > > Meaning Maximum size of all memstores in the region server before new
> > > > updates
> > > > are blocked and flushes are forced is 7352m which is lower than 774m.
> > > >
> > > > You can increase the value for
> hbase.regionserver.global.memstore.size
> > > >
> > > > Please also see if you can distribute the writes to the underlying
> > region
> > > > so that the region's use of memstore comes down.
> > > >
> > > > Cheersx
> > > >
> > > > On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang <
> > jianshi.huang@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > Only one region server is problematic.
> > > > >
> > > > > hbase.regionserver.global.memstore.size is not set, the problematic
> > > > region
> > > > > is using 774m for memstore.
> > > > >
> > > > > Max heap is 18380m for all region servers.
> > > > >
> > > > > Jianshi
> > > > >
> > > > >
> > > > > On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > What's the value for hbase.regionserver.global.memstore.size ?
> > > > > >
> > > > > > Did RegionTooBusyException happen to many regions or only a few
> > > > regions ?
> > > > > >
> > > > > > How much heap did you give region servers ?
> > > > > >
> > > > > > bq. HBase version is 0.98.0.2.1.2.0-402
> > > > > >
> > > > > > Yeah, this is a bit old. Please consider upgrading.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <
> > > jianshi.huang@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm constantly facing "RegionTooBusyException: Above memstore
> > > limit"
> > > > > > errors
> > > > > > > in one region server when writing data to HBase.
> > > > > > >
> > > > > > > I checked the region server log, and I've seen a lot of
> warnings
> > > > during
> > > > > > the
> > > > > > > data writes:
> > > > > > >
> > > > > > > WARN wal.fshlog couldn't find oldest seqNum for the region
> > we're
> > > > > about
> > > > > > to
> > > > > > > flush, ...
> > > > > > >
> > > > > > > Then HBase seem to flush the data and added it as a HStore
> file.
> > > > > > >
> > > > > > > I also get a few warnings in client.ShortCircuitCache, says
> > "could
> > > > not
> > > > > > load
> > > > > > > ... due to InvalidToken exceptions.
> > > > > > >
> > > > > > > Anyone can give me hint what went wrong?
> > > > > > >
> > > > > > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but
> > the
> > > > > > release
> > > > > > > is a little bit old.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Jianshi Huang
> > > > > > >
> > > > > > > LinkedIn: jianshi
> > > > > > > Twitter: @jshuang
> > > > > > > Github & Blog: http://huangjs.github.com/
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Jianshi Huang
> > > > >
> > > > > LinkedIn: jianshi
> > > > > Twitter: @jshuang
> > > > > Github & Blog: http://huangjs.github.com/
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
It depends on how you manage your connection, your table and your puts. If
it works for you with reducing the batch buffer size, then just keep it the
way it is...
JM
2015-03-03 11:10 GMT-05:00 Jianshi Huang <ji...@gmail.com>:
> Yes, looks like reducing the batch buffer size works (still validating).
>
> But why setAutoFlush(false) is harmful here? I just want maximum write
> speed.
>
> Jianshi
>
> On Tue, Mar 3, 2015 at 10:54 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Let HBase manage the flushes for you. Remove
> edgeTable.setAutoFlush(false)
> > and maybe reduce your batch size.
> >
> > I don't think that increasing the memstore is the good way to go. Sound
> > more like a plaster on the issue than a good fix (for me).
> >
> > JM
> >
> > 2015-03-03 9:43 GMT-05:00 Ted Yu <yu...@gmail.com>:
> >
> > > Default value for hbase.regionserver.global.memstore.size is 0.4
> > >
> > > Meaning Maximum size of all memstores in the region server before new
> > > updates
> > > are blocked and flushes are forced is 7352m which is lower than 774m.
> > >
> > > You can increase the value for hbase.regionserver.global.memstore.size
> > >
> > > Please also see if you can distribute the writes to the underlying
> region
> > > so that the region's use of memstore comes down.
> > >
> > > Cheersx
> > >
> > > On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang <
> jianshi.huang@gmail.com>
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > Only one region server is problematic.
> > > >
> > > > hbase.regionserver.global.memstore.size is not set, the problematic
> > > region
> > > > is using 774m for memstore.
> > > >
> > > > Max heap is 18380m for all region servers.
> > > >
> > > > Jianshi
> > > >
> > > >
> > > > On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > What's the value for hbase.regionserver.global.memstore.size ?
> > > > >
> > > > > Did RegionTooBusyException happen to many regions or only a few
> > > regions ?
> > > > >
> > > > > How much heap did you give region servers ?
> > > > >
> > > > > bq. HBase version is 0.98.0.2.1.2.0-402
> > > > >
> > > > > Yeah, this is a bit old. Please consider upgrading.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <
> > jianshi.huang@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm constantly facing "RegionTooBusyException: Above memstore
> > limit"
> > > > > errors
> > > > > > in one region server when writing data to HBase.
> > > > > >
> > > > > > I checked the region server log, and I've seen a lot of warnings
> > > during
> > > > > the
> > > > > > data writes:
> > > > > >
> > > > > > WARN wal.fshlog couldn't find oldest seqNum for the region
> we're
> > > > about
> > > > > to
> > > > > > flush, ...
> > > > > >
> > > > > > Then HBase seem to flush the data and added it as a HStore file.
> > > > > >
> > > > > > I also get a few warnings in client.ShortCircuitCache, says
> "could
> > > not
> > > > > load
> > > > > > ... due to InvalidToken exceptions.
> > > > > >
> > > > > > Anyone can give me hint what went wrong?
> > > > > >
> > > > > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but
> the
> > > > > release
> > > > > > is a little bit old.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Jianshi Huang
> > > > > >
> > > > > > LinkedIn: jianshi
> > > > > > Twitter: @jshuang
> > > > > > Github & Blog: http://huangjs.github.com/
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Jianshi Huang
> > > >
> > > > LinkedIn: jianshi
> > > > Twitter: @jshuang
> > > > Github & Blog: http://huangjs.github.com/
> > > >
> > >
> >
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jianshi Huang <ji...@gmail.com>.
Yes, looks like reducing the batch buffer size works (still validating).
But why setAutoFlush(false) is harmful here? I just want maximum write
speed.
Jianshi
On Tue, Mar 3, 2015 at 10:54 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Let HBase manage the flushes for you. Remove edgeTable.setAutoFlush(false)
> and maybe reduce your batch size.
>
> I don't think that increasing the memstore is the good way to go. Sound
> more like a plaster on the issue than a good fix (for me).
>
> JM
>
> 2015-03-03 9:43 GMT-05:00 Ted Yu <yu...@gmail.com>:
>
> > Default value for hbase.regionserver.global.memstore.size is 0.4
> >
> > Meaning Maximum size of all memstores in the region server before new
> > updates
> > are blocked and flushes are forced is 7352m which is lower than 774m.
> >
> > You can increase the value for hbase.regionserver.global.memstore.size
> >
> > Please also see if you can distribute the writes to the underlying region
> > so that the region's use of memstore comes down.
> >
> > Cheersx
> >
> > On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang <ji...@gmail.com>
> > wrote:
> >
> > > Hi Ted,
> > >
> > > Only one region server is problematic.
> > >
> > > hbase.regionserver.global.memstore.size is not set, the problematic
> > region
> > > is using 774m for memstore.
> > >
> > > Max heap is 18380m for all region servers.
> > >
> > > Jianshi
> > >
> > >
> > > On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > What's the value for hbase.regionserver.global.memstore.size ?
> > > >
> > > > Did RegionTooBusyException happen to many regions or only a few
> > regions ?
> > > >
> > > > How much heap did you give region servers ?
> > > >
> > > > bq. HBase version is 0.98.0.2.1.2.0-402
> > > >
> > > > Yeah, this is a bit old. Please consider upgrading.
> > > >
> > > > Cheers
> > > >
> > > > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <
> jianshi.huang@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm constantly facing "RegionTooBusyException: Above memstore
> limit"
> > > > errors
> > > > > in one region server when writing data to HBase.
> > > > >
> > > > > I checked the region server log, and I've seen a lot of warnings
> > during
> > > > the
> > > > > data writes:
> > > > >
> > > > > WARN wal.fshlog couldn't find oldest seqNum for the region we're
> > > about
> > > > to
> > > > > flush, ...
> > > > >
> > > > > Then HBase seem to flush the data and added it as a HStore file.
> > > > >
> > > > > I also get a few warnings in client.ShortCircuitCache, says "could
> > not
> > > > load
> > > > > ... due to InvalidToken exceptions.
> > > > >
> > > > > Anyone can give me hint what went wrong?
> > > > >
> > > > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the
> > > > release
> > > > > is a little bit old.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Jianshi Huang
> > > > >
> > > > > LinkedIn: jianshi
> > > > > Twitter: @jshuang
> > > > > Github & Blog: http://huangjs.github.com/
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Let HBase manage the flushes for you. Remove edgeTable.setAutoFlush(false)
and maybe reduce your batch size.
I don't think that increasing the memstore is the good way to go. Sound
more like a plaster on the issue than a good fix (for me).
JM
2015-03-03 9:43 GMT-05:00 Ted Yu <yu...@gmail.com>:
> Default value for hbase.regionserver.global.memstore.size is 0.4
>
> Meaning Maximum size of all memstores in the region server before new
> updates
> are blocked and flushes are forced is 7352m which is lower than 774m.
>
> You can increase the value for hbase.regionserver.global.memstore.size
>
> Please also see if you can distribute the writes to the underlying region
> so that the region's use of memstore comes down.
>
> Cheersx
>
> On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > Hi Ted,
> >
> > Only one region server is problematic.
> >
> > hbase.regionserver.global.memstore.size is not set, the problematic
> region
> > is using 774m for memstore.
> >
> > Max heap is 18380m for all region servers.
> >
> > Jianshi
> >
> >
> > On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > What's the value for hbase.regionserver.global.memstore.size ?
> > >
> > > Did RegionTooBusyException happen to many regions or only a few
> regions ?
> > >
> > > How much heap did you give region servers ?
> > >
> > > bq. HBase version is 0.98.0.2.1.2.0-402
> > >
> > > Yeah, this is a bit old. Please consider upgrading.
> > >
> > > Cheers
> > >
> > > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <jianshi.huang@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm constantly facing "RegionTooBusyException: Above memstore limit"
> > > errors
> > > > in one region server when writing data to HBase.
> > > >
> > > > I checked the region server log, and I've seen a lot of warnings
> during
> > > the
> > > > data writes:
> > > >
> > > > WARN wal.fshlog couldn't find oldest seqNum for the region we're
> > about
> > > to
> > > > flush, ...
> > > >
> > > > Then HBase seem to flush the data and added it as a HStore file.
> > > >
> > > > I also get a few warnings in client.ShortCircuitCache, says "could
> not
> > > load
> > > > ... due to InvalidToken exceptions.
> > > >
> > > > Anyone can give me hint what went wrong?
> > > >
> > > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the
> > > release
> > > > is a little bit old.
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Jianshi Huang
> > > >
> > > > LinkedIn: jianshi
> > > > Twitter: @jshuang
> > > > Github & Blog: http://huangjs.github.com/
> > > >
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Ted Yu <yu...@gmail.com>.
Default value for hbase.regionserver.global.memstore.size is 0.4
Meaning Maximum size of all memstores in the region server before new updates
are blocked and flushes are forced is 7352m which is lower than 774m.
You can increase the value for hbase.regionserver.global.memstore.size
Please also see if you can distribute the writes to the underlying region
so that the region's use of memstore comes down.
Cheersx
On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang <ji...@gmail.com>
wrote:
> Hi Ted,
>
> Only one region server is problematic.
>
> hbase.regionserver.global.memstore.size is not set, the problematic region
> is using 774m for memstore.
>
> Max heap is 18380m for all region servers.
>
> Jianshi
>
>
> On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > What's the value for hbase.regionserver.global.memstore.size ?
> >
> > Did RegionTooBusyException happen to many regions or only a few regions ?
> >
> > How much heap did you give region servers ?
> >
> > bq. HBase version is 0.98.0.2.1.2.0-402
> >
> > Yeah, this is a bit old. Please consider upgrading.
> >
> > Cheers
> >
> > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <ji...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I'm constantly facing "RegionTooBusyException: Above memstore limit"
> > errors
> > > in one region server when writing data to HBase.
> > >
> > > I checked the region server log, and I've seen a lot of warnings during
> > the
> > > data writes:
> > >
> > > WARN wal.fshlog couldn't find oldest seqNum for the region we're
> about
> > to
> > > flush, ...
> > >
> > > Then HBase seem to flush the data and added it as a HStore file.
> > >
> > > I also get a few warnings in client.ShortCircuitCache, says "could not
> > load
> > > ... due to InvalidToken exceptions.
> > >
> > > Anyone can give me hint what went wrong?
> > >
> > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the
> > release
> > > is a little bit old.
> > >
> > > Thanks,
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
Problem: about hbase.regionserver.restart.on.zk.expire configuration
Posted by "Yu, Bella" <Be...@Honeywell.com>.
Hi Ted,
Now we found in version of HBASE v0.98.5, there can't be set the configuration about "hbase.regionserver.restart.on.zk.expire" ,so would you please tell us there have other ways to replace this configuration in v0.98.5.
thanks
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jianshi Huang <ji...@gmail.com>.
Hi Ted,
Only one region server is problematic.
hbase.regionserver.global.memstore.size is not set, the problematic region
is using 774m for memstore.
Max heap is 18380m for all region servers.
Jianshi
On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <yu...@gmail.com> wrote:
> What's the value for hbase.regionserver.global.memstore.size ?
>
> Did RegionTooBusyException happen to many regions or only a few regions ?
>
> How much heap did you give region servers ?
>
> bq. HBase version is 0.98.0.2.1.2.0-402
>
> Yeah, this is a bit old. Please consider upgrading.
>
> Cheers
>
> On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm constantly facing "RegionTooBusyException: Above memstore limit"
> errors
> > in one region server when writing data to HBase.
> >
> > I checked the region server log, and I've seen a lot of warnings during
> the
> > data writes:
> >
> > WARN wal.fshlog couldn't find oldest seqNum for the region we're about
> to
> > flush, ...
> >
> > Then HBase seem to flush the data and added it as a HStore file.
> >
> > I also get a few warnings in client.ShortCircuitCache, says "could not
> load
> > ... due to InvalidToken exceptions.
> >
> > Anyone can give me hint what went wrong?
> >
> > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the
> release
> > is a little bit old.
> >
> > Thanks,
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jianshi Huang <ji...@gmail.com>.
Hi JM,
Thanks for the hints. Here's my settings for writer.
edgeTable.setAutoFlush(false)
edgeTable.setWriteBufferSize(20971520)
The write buffer seems quite large as the region server is hosting 12
related regions I'm writing to. I'll test with smaller write buffer size.
The size of each put is between 10k~100k.
Jianshi
On Mon, Mar 2, 2015 at 11:04 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Jo Jianshi,
>
> Are you doing batch of puts? If so, what's the size of the batch and what's
> the size of the puts? Are you trying to give a batch which at the end will
> be bigger than the memstore size for a single RS? Can you try to reduce the
> size of this batch?
>
> JM
>
> 2015-03-02 9:59 GMT-05:00 Ted Yu <yu...@gmail.com>:
>
> > What's the value for hbase.regionserver.global.memstore.size ?
> >
> > Did RegionTooBusyException happen to many regions or only a few regions ?
> >
> > How much heap did you give region servers ?
> >
> > bq. HBase version is 0.98.0.2.1.2.0-402
> >
> > Yeah, this is a bit old. Please consider upgrading.
> >
> > Cheers
> >
> > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <ji...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I'm constantly facing "RegionTooBusyException: Above memstore limit"
> > errors
> > > in one region server when writing data to HBase.
> > >
> > > I checked the region server log, and I've seen a lot of warnings during
> > the
> > > data writes:
> > >
> > > WARN wal.fshlog couldn't find oldest seqNum for the region we're
> about
> > to
> > > flush, ...
> > >
> > > Then HBase seem to flush the data and added it as a HStore file.
> > >
> > > I also get a few warnings in client.ShortCircuitCache, says "could not
> > load
> > > ... due to InvalidToken exceptions.
> > >
> > > Anyone can give me hint what went wrong?
> > >
> > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the
> > release
> > > is a little bit old.
> > >
> > > Thanks,
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Jo Jianshi,
Are you doing batch of puts? If so, what's the size of the batch and what's
the size of the puts? Are you trying to give a batch which at the end will
be bigger than the memstore size for a single RS? Can you try to reduce the
size of this batch?
JM
2015-03-02 9:59 GMT-05:00 Ted Yu <yu...@gmail.com>:
> What's the value for hbase.regionserver.global.memstore.size ?
>
> Did RegionTooBusyException happen to many regions or only a few regions ?
>
> How much heap did you give region servers ?
>
> bq. HBase version is 0.98.0.2.1.2.0-402
>
> Yeah, this is a bit old. Please consider upgrading.
>
> Cheers
>
> On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm constantly facing "RegionTooBusyException: Above memstore limit"
> errors
> > in one region server when writing data to HBase.
> >
> > I checked the region server log, and I've seen a lot of warnings during
> the
> > data writes:
> >
> > WARN wal.fshlog couldn't find oldest seqNum for the region we're about
> to
> > flush, ...
> >
> > Then HBase seem to flush the data and added it as a HStore file.
> >
> > I also get a few warnings in client.ShortCircuitCache, says "could not
> load
> > ... due to InvalidToken exceptions.
> >
> > Anyone can give me hint what went wrong?
> >
> > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the
> release
> > is a little bit old.
> >
> > Thanks,
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
Re: Need Help: RegionTooBusyException: Above memstore limit
Posted by Ted Yu <yu...@gmail.com>.
What's the value for hbase.regionserver.global.memstore.size ?
Did RegionTooBusyException happen to many regions or only a few regions ?
How much heap did you give region servers ?
bq. HBase version is 0.98.0.2.1.2.0-402
Yeah, this is a bit old. Please consider upgrading.
Cheers
On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang <ji...@gmail.com>
wrote:
> Hi,
>
> I'm constantly facing "RegionTooBusyException: Above memstore limit" errors
> in one region server when writing data to HBase.
>
> I checked the region server log, and I've seen a lot of warnings during the
> data writes:
>
> WARN wal.fshlog couldn't find oldest seqNum for the region we're about to
> flush, ...
>
> Then HBase seem to flush the data and added it as a HStore file.
>
> I also get a few warnings in client.ShortCircuitCache, says "could not load
> ... due to InvalidToken exceptions.
>
> Anyone can give me hint what went wrong?
>
> My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but the release
> is a little bit old.
>
> Thanks,
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>