You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by ozhang <oz...@gmail.com> on 2014/08/03 15:37:58 UTC
hbase set memstore size to 0
Hello,
In our hbase cluster memstore flush size is 128 mb. And to insert data to
tables, we only use bulk load tool. Since bulk loading bypasses memstores,
they are never used, so we want to minimize memstore flush size. But
memstore flush size is used in many important calculations in hbase such
that;
region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
“hbase.hregion.max.filesize”)
So setting memstore value smaller or "0" for example, results in some other
problems.
What do you suggest us in that case. Setting memstore size to 128 holds some
memory for tens of regions in region server and we want to get rid of it.
Thanks a lot.
ozhan
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: hbase set memstore size to 0
Posted by ozhang <oz...@gmail.com>.
I check that and see that memstore global upper limit is 0.4. As you
suggested setting it to a lower value will to what i want. Also
ConstantSizeRegionSplitPolicy is a nice suggestion. Thanks you so much.
Best regards.
Ozhan
3 Ağu 2014 22:49 tarihinde "Jean-Marc Spaggiari [via Apache HBase]" <
ml-node+s679495n4062263h19@n3.nabble.com> yazdı:
> Just to add to Kevin's comments. Are you looking at that because of the
> split policy? If that's the case, can you not just change the policy? Like
> configure your table to use the ConstantSizeRegionSplitPolicy?
>
>
> 2014-08-03 15:15 GMT-04:00 Kevin O'dell <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=4062263&i=0>>:
>
> > Upon insert, lower the global setting not the flush size :)
> > On Aug 3, 2014 3:01 PM, "ozhang" <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=4062263&i=1>> wrote:
> >
> > > Hi Kevin,
> > > We guess that, on region server start up, hbase gets some memory for
> each
> > > memstore. So we want to decrease this value. You are saying that
> memstore
> > > size doesnt matter. If memstore flush size 128 mb, does java take some
> > > memory for each memstore on region startup. Or it only takes memory
> while
> > > you are using it to insert data.
> > > Thanks a lot
> > > 3 Ağu 2014 21:27 tarihinde "Kevin O'dell [via Apache HBase]" <
> > > [hidden email] <http://user/SendEmail.jtp?type=node&node=4062263&i=2>>
> yazdı:
> > >
> > > > Hi Ozhang,
> > > >
> > > > If you are only bulk loading into HBase, then memstore flush size
> > > should
> > > > not matter. You most likely you looking to lower the upper/global
> > > memstore
> > > > limits.
> > > > On Aug 3, 2014 2:23 PM, "ozhang" <[hidden email]
> > > > <http://user/SendEmail.jtp?type=node&node=4062260&i=0>> wrote:
> > > >
> > > > > Hello,
> > > > > In our hbase cluster memstore flush size is 128 mb. And to insert
> > data
> > > > to
> > > > > tables, we only use bulk load tool. Since bulk loading bypasses
> > > > memstores,
> > > > > they are never used, so we want to minimize memstore flush size.
> But
> > > > > memstore flush size is used in many important calculations in
> hbase
> > > such
> > > > > that;
> > > > >
> > > > > region split size = Min (R^2 *
> “hbase.hregion.memstore.flush.size”,
> > > > > “hbase.hregion.max.filesize”)
> > > > >
> > > > > So setting memstore value smaller or "0" for example, results in
> > some
> > > > > other
> > > > > problems.
> > > > > What do you suggest us in that case. Setting memstore size to 128
> > holds
> > > > > some
> > > > > memory for tens of regions in region server and we want to get rid
> of
> > > > it.
> > > > > Thanks a lot.
> > > > >
> > > > > ozhan
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > View this message in context:
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
> > > > > Sent from the HBase User mailing list archive at Nabble.com.
> > > > >
> > > >
> > > >
> > > > ------------------------------
> > > > If you reply to this email, your message will be added to the
> > discussion
> > > > below:
> > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062260.html
> > > > To unsubscribe from hbase set memstore size to 0, click here
> > > > <
> > >
> > > >
> > > > .
> > > > NAML
> > > > <
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062261.html
> > > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062263.html
> To unsubscribe from hbase set memstore size to 0, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4062257&code=b3poYW5ndWxlbkBnbWFpbC5jb218NDA2MjI1N3wxMDY0OTA3MTE4>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062264.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: hbase set memstore size to 0
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Just to add to Kevin's comments. Are you looking at that because of the
split policy? If that's the case, can you not just change the policy? Like
configure your table to use the ConstantSizeRegionSplitPolicy?
2014-08-03 15:15 GMT-04:00 Kevin O'dell <ke...@cloudera.com>:
> Upon insert, lower the global setting not the flush size :)
> On Aug 3, 2014 3:01 PM, "ozhang" <oz...@gmail.com> wrote:
>
> > Hi Kevin,
> > We guess that, on region server start up, hbase gets some memory for each
> > memstore. So we want to decrease this value. You are saying that memstore
> > size doesnt matter. If memstore flush size 128 mb, does java take some
> > memory for each memstore on region startup. Or it only takes memory while
> > you are using it to insert data.
> > Thanks a lot
> > 3 Ağu 2014 21:27 tarihinde "Kevin O'dell [via Apache HBase]" <
> > ml-node+s679495n4062260h26@n3.nabble.com> yazdı:
> >
> > > Hi Ozhang,
> > >
> > > If you are only bulk loading into HBase, then memstore flush size
> > should
> > > not matter. You most likely you looking to lower the upper/global
> > memstore
> > > limits.
> > > On Aug 3, 2014 2:23 PM, "ozhang" <[hidden email]
> > > <http://user/SendEmail.jtp?type=node&node=4062260&i=0>> wrote:
> > >
> > > > Hello,
> > > > In our hbase cluster memstore flush size is 128 mb. And to insert
> data
> > > to
> > > > tables, we only use bulk load tool. Since bulk loading bypasses
> > > memstores,
> > > > they are never used, so we want to minimize memstore flush size. But
> > > > memstore flush size is used in many important calculations in hbase
> > such
> > > > that;
> > > >
> > > > region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
> > > > “hbase.hregion.max.filesize”)
> > > >
> > > > So setting memstore value smaller or "0" for example, results in
> some
> > > > other
> > > > problems.
> > > > What do you suggest us in that case. Setting memstore size to 128
> holds
> > > > some
> > > > memory for tens of regions in region server and we want to get rid of
> > > it.
> > > > Thanks a lot.
> > > >
> > > > ozhan
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
> > > > Sent from the HBase User mailing list archive at Nabble.com.
> > > >
> > >
> > >
> > > ------------------------------
> > > If you reply to this email, your message will be added to the
> discussion
> > > below:
> > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062260.html
> > > To unsubscribe from hbase set memstore size to 0, click here
> > > <
> >
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4062257&code=b3poYW5ndWxlbkBnbWFpbC5jb218NDA2MjI1N3wxMDY0OTA3MTE4
> > >
> > > .
> > > NAML
> > > <
> >
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> > >
> > >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062261.html
> > Sent from the HBase User mailing list archive at Nabble.com.
>
Re: hbase set memstore size to 0
Posted by Kevin O'dell <ke...@cloudera.com>.
Upon insert, lower the global setting not the flush size :)
On Aug 3, 2014 3:01 PM, "ozhang" <oz...@gmail.com> wrote:
> Hi Kevin,
> We guess that, on region server start up, hbase gets some memory for each
> memstore. So we want to decrease this value. You are saying that memstore
> size doesnt matter. If memstore flush size 128 mb, does java take some
> memory for each memstore on region startup. Or it only takes memory while
> you are using it to insert data.
> Thanks a lot
> 3 Ağu 2014 21:27 tarihinde "Kevin O'dell [via Apache HBase]" <
> ml-node+s679495n4062260h26@n3.nabble.com> yazdı:
>
> > Hi Ozhang,
> >
> > If you are only bulk loading into HBase, then memstore flush size
> should
> > not matter. You most likely you looking to lower the upper/global
> memstore
> > limits.
> > On Aug 3, 2014 2:23 PM, "ozhang" <[hidden email]
> > <http://user/SendEmail.jtp?type=node&node=4062260&i=0>> wrote:
> >
> > > Hello,
> > > In our hbase cluster memstore flush size is 128 mb. And to insert data
> > to
> > > tables, we only use bulk load tool. Since bulk loading bypasses
> > memstores,
> > > they are never used, so we want to minimize memstore flush size. But
> > > memstore flush size is used in many important calculations in hbase
> such
> > > that;
> > >
> > > region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
> > > “hbase.hregion.max.filesize”)
> > >
> > > So setting memstore value smaller or "0" for example, results in some
> > > other
> > > problems.
> > > What do you suggest us in that case. Setting memstore size to 128 holds
> > > some
> > > memory for tens of regions in region server and we want to get rid of
> > it.
> > > Thanks a lot.
> > >
> > > ozhan
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
> > > Sent from the HBase User mailing list archive at Nabble.com.
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062260.html
> > To unsubscribe from hbase set memstore size to 0, click here
> > <
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4062257&code=b3poYW5ndWxlbkBnbWFpbC5jb218NDA2MjI1N3wxMDY0OTA3MTE4
> >
> > .
> > NAML
> > <
> http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062261.html
> Sent from the HBase User mailing list archive at Nabble.com.
Re: hbase set memstore size to 0
Posted by ozhang <oz...@gmail.com>.
Hi Kevin,
We guess that, on region server start up, hbase gets some memory for each
memstore. So we want to decrease this value. You are saying that memstore
size doesnt matter. If memstore flush size 128 mb, does java take some
memory for each memstore on region startup. Or it only takes memory while
you are using it to insert data.
Thanks a lot
3 Ağu 2014 21:27 tarihinde "Kevin O'dell [via Apache HBase]" <
ml-node+s679495n4062260h26@n3.nabble.com> yazdı:
> Hi Ozhang,
>
> If you are only bulk loading into HBase, then memstore flush size should
> not matter. You most likely you looking to lower the upper/global memstore
> limits.
> On Aug 3, 2014 2:23 PM, "ozhang" <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=4062260&i=0>> wrote:
>
> > Hello,
> > In our hbase cluster memstore flush size is 128 mb. And to insert data
> to
> > tables, we only use bulk load tool. Since bulk loading bypasses
> memstores,
> > they are never used, so we want to minimize memstore flush size. But
> > memstore flush size is used in many important calculations in hbase such
> > that;
> >
> > region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
> > “hbase.hregion.max.filesize”)
> >
> > So setting memstore value smaller or "0" for example, results in some
> > other
> > problems.
> > What do you suggest us in that case. Setting memstore size to 128 holds
> > some
> > memory for tens of regions in region server and we want to get rid of
> it.
> > Thanks a lot.
> >
> > ozhan
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062260.html
> To unsubscribe from hbase set memstore size to 0, click here
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4062257&code=b3poYW5ndWxlbkBnbWFpbC5jb218NDA2MjI1N3wxMDY0OTA3MTE4>
> .
> NAML
> <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062261.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: hbase set memstore size to 0
Posted by Kevin O'dell <ke...@cloudera.com>.
Hi Ozhang,
If you are only bulk loading into HBase, then memstore flush size should
not matter. You most likely you looking to lower the upper/global memstore
limits.
On Aug 3, 2014 2:23 PM, "ozhang" <oz...@gmail.com> wrote:
> Hello,
> In our hbase cluster memstore flush size is 128 mb. And to insert data to
> tables, we only use bulk load tool. Since bulk loading bypasses memstores,
> they are never used, so we want to minimize memstore flush size. But
> memstore flush size is used in many important calculations in hbase such
> that;
>
> region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
> “hbase.hregion.max.filesize”)
>
> So setting memstore value smaller or "0" for example, results in some
> other
> problems.
> What do you suggest us in that case. Setting memstore size to 128 holds
> some
> memory for tens of regions in region server and we want to get rid of it.
> Thanks a lot.
>
> ozhan
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
> Sent from the HBase User mailing list archive at Nabble.com.
>