You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Alexander Ilyin <al...@weborama.com> on 2016/10/17 15:26:05 UTC
HBase restart without region reassigning
Hi,
We have a 7-node HBase cluster (version 1.1.2) and we change some of its
settings from time to time which requires a restart. The problem is that
every time after the restart load balancer reassigns the regions making
data locality low.
To address this issue we tried the settings described here:
https://issues.apache.org/jira/browse/HBASE-6389,
"hbase.master.wait.on.regionservers.interval"
in particular. We tried it two times in slightly different ways but neither
of them worked. First time we did a rolling restart (master, then each of
datanodes) and we saw 14 datanodes instead of 7 in Master UI. Half of them
had the regions on it while the other half was empty. We restarted master
only and we got 7 empty datanodes in Master UI. After that we rollbacked
the setting.
Second time we restarted master and datanodes at the same time but master
failed to read meta table, moved it to a different datanode and reassigned
the regions again.
Please advise on how to use hbase.master.wait.on.regionservers.* settings
properly. Launching major compactions for all the tables after each config
change seems to be an overkill. Attaching Master server logs with relevant
lines for two attempts mentioned above.
Thanks in advance.
Re: HBase restart without region reassigning
Posted by Ted Yu <yu...@gmail.com>.
Since you are using 1.1.2, you may want to look at HBASE-14531 which was
fixed in 1.1.3
FYI
On Mon, Oct 17, 2016 at 9:13 AM, Alexander Ilyin <al...@weborama.com>
wrote:
> I'm restarting it through Ambari. First time I specified a delay between
> regionserver restarts, second time I didn't. Not sure whether Ambari uses
> graceful restart script internally but I can try to use it directly.
>
> On Mon, Oct 17, 2016 at 6:00 PM, Jeremy Carroll <ph...@gmail.com>
> wrote:
>
> > How are you restarting the cluster? From my experience a graceful rolling
> > restart retains locality.
> >
> > For each regionserver (one at a time) run the graceful restart script to
> > retain local blocks. The master configuration option you specified only
> > works on a full cluster reboot (or master reboot)
> >
> > On Mon, Oct 17, 2016 at 8:26 AM Alexander Ilyin <al...@weborama.com>
> > wrote:
> >
> > > Hi,
> > >
> > > We have a 7-node HBase cluster (version 1.1.2) and we change some of
> its
> > > settings from time to time which requires a restart. The problem is
> that
> > > every time after the restart load balancer reassigns the regions making
> > > data locality low.
> > >
> > > To address this issue we tried the settings described here:
> > > https://issues.apache.org/jira/browse/HBASE-6389,
> > > "hbase.master.wait.on.regionservers.interval" in particular. We tried
> it
> > > two times in slightly different ways but neither of them worked. First
> > time
> > > we did a rolling restart (master, then each of datanodes) and we saw 14
> > > datanodes instead of 7 in Master UI. Half of them had the regions on it
> > > while the other half was empty. We restarted master only and we got 7
> > empty
> > > datanodes in Master UI. After that we rollbacked the setting.
> > >
> > > Second time we restarted master and datanodes at the same time but
> master
> > > failed to read meta table, moved it to a different datanode and
> > reassigned
> > > the regions again.
> > >
> > > Please advise on how to use hbase.master.wait.on.regionservers.*
> > settings
> > > properly. Launching major compactions for all the tables after each
> > config
> > > change seems to be an overkill. Attaching Master server logs with
> > relevant
> > > lines for two attempts mentioned above.
> > >
> > > Thanks in advance.
> > >
> >
>
Re: HBase restart without region reassigning
Posted by Alexander Ilyin <al...@weborama.com>.
I'm restarting it through Ambari. First time I specified a delay between
regionserver restarts, second time I didn't. Not sure whether Ambari uses
graceful restart script internally but I can try to use it directly.
On Mon, Oct 17, 2016 at 6:00 PM, Jeremy Carroll <ph...@gmail.com> wrote:
> How are you restarting the cluster? From my experience a graceful rolling
> restart retains locality.
>
> For each regionserver (one at a time) run the graceful restart script to
> retain local blocks. The master configuration option you specified only
> works on a full cluster reboot (or master reboot)
>
> On Mon, Oct 17, 2016 at 8:26 AM Alexander Ilyin <al...@weborama.com>
> wrote:
>
> > Hi,
> >
> > We have a 7-node HBase cluster (version 1.1.2) and we change some of its
> > settings from time to time which requires a restart. The problem is that
> > every time after the restart load balancer reassigns the regions making
> > data locality low.
> >
> > To address this issue we tried the settings described here:
> > https://issues.apache.org/jira/browse/HBASE-6389,
> > "hbase.master.wait.on.regionservers.interval" in particular. We tried it
> > two times in slightly different ways but neither of them worked. First
> time
> > we did a rolling restart (master, then each of datanodes) and we saw 14
> > datanodes instead of 7 in Master UI. Half of them had the regions on it
> > while the other half was empty. We restarted master only and we got 7
> empty
> > datanodes in Master UI. After that we rollbacked the setting.
> >
> > Second time we restarted master and datanodes at the same time but master
> > failed to read meta table, moved it to a different datanode and
> reassigned
> > the regions again.
> >
> > Please advise on how to use hbase.master.wait.on.regionservers.*
> settings
> > properly. Launching major compactions for all the tables after each
> config
> > change seems to be an overkill. Attaching Master server logs with
> relevant
> > lines for two attempts mentioned above.
> >
> > Thanks in advance.
> >
>
Re: HBase restart without region reassigning
Posted by Jeremy Carroll <ph...@gmail.com>.
How are you restarting the cluster? From my experience a graceful rolling
restart retains locality.
For each regionserver (one at a time) run the graceful restart script to
retain local blocks. The master configuration option you specified only
works on a full cluster reboot (or master reboot)
On Mon, Oct 17, 2016 at 8:26 AM Alexander Ilyin <al...@weborama.com>
wrote:
> Hi,
>
> We have a 7-node HBase cluster (version 1.1.2) and we change some of its
> settings from time to time which requires a restart. The problem is that
> every time after the restart load balancer reassigns the regions making
> data locality low.
>
> To address this issue we tried the settings described here:
> https://issues.apache.org/jira/browse/HBASE-6389,
> "hbase.master.wait.on.regionservers.interval" in particular. We tried it
> two times in slightly different ways but neither of them worked. First time
> we did a rolling restart (master, then each of datanodes) and we saw 14
> datanodes instead of 7 in Master UI. Half of them had the regions on it
> while the other half was empty. We restarted master only and we got 7 empty
> datanodes in Master UI. After that we rollbacked the setting.
>
> Second time we restarted master and datanodes at the same time but master
> failed to read meta table, moved it to a different datanode and reassigned
> the regions again.
>
> Please advise on how to use hbase.master.wait.on.regionservers.* settings
> properly. Launching major compactions for all the tables after each config
> change seems to be an overkill. Attaching Master server logs with relevant
> lines for two attempts mentioned above.
>
> Thanks in advance.
>
Re: HBase restart without region reassigning
Posted by Alexander Ilyin <al...@weborama.com>.
Hi Dima,
These are instances in the cloud and we're using Consul for name
resolution. Regarding network settings, your question is a bit broad...
Which settings would you recommend to check first?
On Mon, Oct 17, 2016 at 5:28 PM, Dima Spivak <di...@apache.org> wrote:
> Hey Alexander,
>
> Could something be amiss in your network settings? Seeing phantom datanodes
> could be tripping things up. Are these physical machines or instances in
> the cloud?
>
> On Monday, October 17, 2016, Alexander Ilyin <al...@weborama.com>
> wrote:
>
> > Hi,
> >
> > We have a 7-node HBase cluster (version 1.1.2) and we change some of its
> > settings from time to time which requires a restart. The problem is that
> > every time after the restart load balancer reassigns the regions making
> > data locality low.
> >
> > To address this issue we tried the settings described here:
> > https://issues.apache.org/jira/browse/HBASE-6389,
> > "hbase.master.wait.on.regionservers.interval" in particular. We tried it
> > two times in slightly different ways but neither of them worked. First
> time
> > we did a rolling restart (master, then each of datanodes) and we saw 14
> > datanodes instead of 7 in Master UI. Half of them had the regions on it
> > while the other half was empty. We restarted master only and we got 7
> empty
> > datanodes in Master UI. After that we rollbacked the setting.
> >
> > Second time we restarted master and datanodes at the same time but master
> > failed to read meta table, moved it to a different datanode and
> reassigned
> > the regions again.
> >
> > Please advise on how to use hbase.master.wait.on.regionservers.*
> settings
> > properly. Launching major compactions for all the tables after each
> config
> > change seems to be an overkill. Attaching Master server logs with
> relevant
> > lines for two attempts mentioned above.
> >
> > Thanks in advance.
> >
>
>
> --
> -Dima
>
Re: HBase restart without region reassigning
Posted by Dima Spivak <di...@apache.org>.
Hey Alexander,
Could something be amiss in your network settings? Seeing phantom datanodes
could be tripping things up. Are these physical machines or instances in
the cloud?
On Monday, October 17, 2016, Alexander Ilyin <al...@weborama.com> wrote:
> Hi,
>
> We have a 7-node HBase cluster (version 1.1.2) and we change some of its
> settings from time to time which requires a restart. The problem is that
> every time after the restart load balancer reassigns the regions making
> data locality low.
>
> To address this issue we tried the settings described here:
> https://issues.apache.org/jira/browse/HBASE-6389,
> "hbase.master.wait.on.regionservers.interval" in particular. We tried it
> two times in slightly different ways but neither of them worked. First time
> we did a rolling restart (master, then each of datanodes) and we saw 14
> datanodes instead of 7 in Master UI. Half of them had the regions on it
> while the other half was empty. We restarted master only and we got 7 empty
> datanodes in Master UI. After that we rollbacked the setting.
>
> Second time we restarted master and datanodes at the same time but master
> failed to read meta table, moved it to a different datanode and reassigned
> the regions again.
>
> Please advise on how to use hbase.master.wait.on.regionservers.* settings
> properly. Launching major compactions for all the tables after each config
> change seems to be an overkill. Attaching Master server logs with relevant
> lines for two attempts mentioned above.
>
> Thanks in advance.
>
--
-Dima