You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kiran <ki...@gmail.com> on 2013/02/08 06:44:00 UTC
Regions in transition
Dear All,
How to move/reassign regions that are always in transition ? Because of the
cluster is not balancing and regionservers are going down one by one. As
only some regions servers are handling the requests.
Another wierd thing is the region stuck in transition startkey matches with
the region that is assigned. Is that possible to have to have tow regions
with same start keys with one assigned and one not ?
Thanks
Kiran
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
Also, I did try repairHoles, all fix options but in vain. I restarted
master atleast thrice.....
On Fri, Feb 8, 2013 at 11:14 AM, kiran <ki...@gmail.com> wrote:
>
> Dear All,
>
> How to move/reassign regions that are always in transition ? Because of
> the cluster is not balancing and regionservers are going down one by one.
> As only some regions servers are handling the requests.
>
> Another wierd thing is the region stuck in transition startkey matches
> with the region that is assigned. Is that possible to have to have tow
> regions with same start keys with one assigned and one not ?
>
> Thanks
> Kiran
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by ramkrishna vasudevan <ra...@gmail.com>.
What is the version that you are using? You can run the tool called HBCK
which should help you.
But before running this what operation were you trying to make? DISABLED
and then ENABLED a table or it was just normal region assignments while
creating table or
region balancing?
Regard
Ram
On Fri, Feb 8, 2013 at 12:20 PM, kiran <ki...@gmail.com> wrote:
> PENDING_OPEN
>
>
> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar <samar.opensource@gmail.com
> >wrote:
>
> > Can you mention the state of the region.. You can find the details in you
> > master status page
> >
> > On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
> >
> > >We ran the command unassign 'REGIONNAME',true the output is "completed
> in
> > >57 sec", but still region is in transition.
> > >
> > >
> > >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
> > ><sa...@gmail.com>wrote:
> > >
> > >> Was the unassigning successful ?
> > >>
> > >> If not you can force it by - unassign 'REGIONNAME', true
> > >>
> > >> Regards,
> > >> Samar
> > >>
> > >>
> > >>
> > >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
> > >>
> > >> >i issued unassign and close_region both but the region is still in
> > >> >transition also I deleted the .META entry for the region. Do I need
> to
> > >> >restart master ?
> > >> >
> > >> >
> > >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> > >> ><sa...@gmail.com>wrote:
> > >> >
> > >> >> Regions should never overlap
> > >> >> Incase a region is in transition for a long long time, you could
> > >> >>possibly
> > >> >> force Unassign a region , if you are ok to lose the region.
> > >> >> Regards,
> > >> >> Samar
> > >> >>
> > >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
> > >> >>
> > >> >> > are always in transition ? Because of the
> > >> >> >cluster is not balancing and regionservers are going down one by
> > >>one.
> > >> >>As
> > >> >> >only some regions servers are handling the requests.
> > >> >> >
> > >> >> >Another wierd thing is the region stuck in transition startkey
> > >>matches
> > >> >> >with
> > >> >>
> > >> >>
> > >> >>
> > >> >
> > >> >
> > >> >--
> > >> >Thank you
> > >> >Kiran Sarvabhotla
> > >> >
> > >> >-----Even a correct decision is wrong when it is taken late
> > >>
> > >>
> > >>
> > >
> > >
> > >--
> > >Thank you
> > >Kiran Sarvabhotla
> > >
> > >-----Even a correct decision is wrong when it is taken late
> >
> >
> >
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
>
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
I think it might be reason for the splits. From 0.94.0, seems the default
split policy has changes
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.html
On Sat, Feb 9, 2013 at 3:37 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Okie. So if you don't mind can you attach your logs. What is the split
> policy and what is the size of the region?
>
> Its better we take a look at it and solve the problem if it is a kernel
> problem.
>
> Regards
> Ram
>
> On Fri, Feb 8, 2013 at 6:09 PM, kiran <ki...@gmail.com> wrote:
>
> > after some searching i found that main reason for regions with same keys
> is
> > splitting and it is the culprit in our case for inconsistency. But I set
> my
> > filesize to a very large size but i am unsure why still splitting is
> > happening
> >
> >
> > On Fri, Feb 8, 2013 at 1:20 PM, kiran <ki...@gmail.com>
> wrote:
> >
> > > Thanks samar and ramakrishna
> > >
> > > I did use option 2 and deleted /hbase from the zkcli shell. I restarted
> > > the cluster now there are more regions in transition. Will it take some
> > > time to complete since it is a new assignment ?
> > >
> > > My hase version is 0.94.1, and I have a doubt with the property
> > > hbase.hregion.max.filesize. I set it to a very large value of 100GB.
> But
> > > some how table regions are not splitted as per the property.
> > >
> > >
> > > On Fri, Feb 8, 2013 at 12:45 PM, samar kumar <
> samar.opensource@gmail.com
> > >wrote:
> > >
> > >> May be ./bin/hbase hbck -repairHoles could be helpful..
> > >>
> > >>
> > >> On 08/02/13 12:38 PM, "Samir Ahmic" <ah...@gmail.com> wrote:
> > >>
> > >> >Hi, Kiran
> > >> >
> > >> >Welcome to beautiful world of HBase transition states :) . When i
> face
> > >> >RIT issue this are steps that i use to resolve it:
> > >> >
> > >> >1. hbase hbck -fixAssignments (this depends on your version of hbase
> > >> >it can be also just -fix)
> > >> >
> > >> >If you don't have luck with 1. then you will need manual
> intervention:
> > >> >Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
> > >> >(or delete ) /hbase, and then restart cluster.
> > >> >
> > >> >This should help you resolve RITs.
> > >> >
> > >> >Regards
> > >> >
> > >> >
> > >> >
> > >> >On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com>
> > >> wrote:
> > >> >> PENDING_OPEN
> > >> >>
> > >> >>
> > >> >> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar
> > >> >><sa...@gmail.com>wrote:
> > >> >>
> > >> >>> Can you mention the state of the region.. You can find the details
> > in
> > >> >>>you
> > >> >>> master status page
> > >> >>>
> > >> >>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com>
> wrote:
> > >> >>>
> > >> >>> >We ran the command unassign 'REGIONNAME',true the output is
> > >> >>>"completed in
> > >> >>> >57 sec", but still region is in transition.
> > >> >>> >
> > >> >>> >
> > >> >>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
> > >> >>> ><sa...@gmail.com>wrote:
> > >> >>> >
> > >> >>> >> Was the unassigning successful ?
> > >> >>> >>
> > >> >>> >> If not you can force it by - unassign 'REGIONNAME', true
> > >> >>> >>
> > >> >>> >> Regards,
> > >> >>> >> Samar
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com>
> > wrote:
> > >> >>> >>
> > >> >>> >> >i issued unassign and close_region both but the region is
> still
> > in
> > >> >>> >> >transition also I deleted the .META entry for the region. Do I
> > >> >>>need to
> > >> >>> >> >restart master ?
> > >> >>> >> >
> > >> >>> >> >
> > >> >>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> > >> >>> >> ><sa...@gmail.com>wrote:
> > >> >>> >> >
> > >> >>> >> >> Regions should never overlap
> > >> >>> >> >> Incase a region is in transition for a long long time, you
> > could
> > >> >>> >> >>possibly
> > >> >>> >> >> force Unassign a region , if you are ok to lose the region.
> > >> >>> >> >> Regards,
> > >> >>> >> >> Samar
> > >> >>> >> >>
> > >> >>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com>
> > >> >>>wrote:
> > >> >>> >> >>
> > >> >>> >> >> > are always in transition ? Because of the
> > >> >>> >> >> >cluster is not balancing and regionservers are going down
> one
> > >> by
> > >> >>> >>one.
> > >> >>> >> >>As
> > >> >>> >> >> >only some regions servers are handling the requests.
> > >> >>> >> >> >
> > >> >>> >> >> >Another wierd thing is the region stuck in transition
> > startkey
> > >> >>> >>matches
> > >> >>> >> >> >with
> > >> >>> >> >>
> > >> >>> >> >>
> > >> >>> >> >>
> > >> >>> >> >
> > >> >>> >> >
> > >> >>> >> >--
> > >> >>> >> >Thank you
> > >> >>> >> >Kiran Sarvabhotla
> > >> >>> >> >
> > >> >>> >> >-----Even a correct decision is wrong when it is taken late
> > >> >>> >>
> > >> >>> >>
> > >> >>> >>
> > >> >>> >
> > >> >>> >
> > >> >>> >--
> > >> >>> >Thank you
> > >> >>> >Kiran Sarvabhotla
> > >> >>> >
> > >> >>> >-----Even a correct decision is wrong when it is taken late
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Thank you
> > >> >> Kiran Sarvabhotla
> > >> >>
> > >> >> -----Even a correct decision is wrong when it is taken late
> > >>
> > >>
> > >>
> > >
> > >
> > > --
> > > Thank you
> > > Kiran Sarvabhotla
> > >
> > > -----Even a correct decision is wrong when it is taken late
> > >
> > >
> >
> >
> > --
> > Thank you
> > Kiran Sarvabhotla
> >
> > -----Even a correct decision is wrong when it is taken late
> >
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Okie. So if you don't mind can you attach your logs. What is the split
policy and what is the size of the region?
Its better we take a look at it and solve the problem if it is a kernel
problem.
Regards
Ram
On Fri, Feb 8, 2013 at 6:09 PM, kiran <ki...@gmail.com> wrote:
> after some searching i found that main reason for regions with same keys is
> splitting and it is the culprit in our case for inconsistency. But I set my
> filesize to a very large size but i am unsure why still splitting is
> happening
>
>
> On Fri, Feb 8, 2013 at 1:20 PM, kiran <ki...@gmail.com> wrote:
>
> > Thanks samar and ramakrishna
> >
> > I did use option 2 and deleted /hbase from the zkcli shell. I restarted
> > the cluster now there are more regions in transition. Will it take some
> > time to complete since it is a new assignment ?
> >
> > My hase version is 0.94.1, and I have a doubt with the property
> > hbase.hregion.max.filesize. I set it to a very large value of 100GB. But
> > some how table regions are not splitted as per the property.
> >
> >
> > On Fri, Feb 8, 2013 at 12:45 PM, samar kumar <samar.opensource@gmail.com
> >wrote:
> >
> >> May be ./bin/hbase hbck -repairHoles could be helpful..
> >>
> >>
> >> On 08/02/13 12:38 PM, "Samir Ahmic" <ah...@gmail.com> wrote:
> >>
> >> >Hi, Kiran
> >> >
> >> >Welcome to beautiful world of HBase transition states :) . When i face
> >> >RIT issue this are steps that i use to resolve it:
> >> >
> >> >1. hbase hbck -fixAssignments (this depends on your version of hbase
> >> >it can be also just -fix)
> >> >
> >> >If you don't have luck with 1. then you will need manual intervention:
> >> >Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
> >> >(or delete ) /hbase, and then restart cluster.
> >> >
> >> >This should help you resolve RITs.
> >> >
> >> >Regards
> >> >
> >> >
> >> >
> >> >On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com>
> >> wrote:
> >> >> PENDING_OPEN
> >> >>
> >> >>
> >> >> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar
> >> >><sa...@gmail.com>wrote:
> >> >>
> >> >>> Can you mention the state of the region.. You can find the details
> in
> >> >>>you
> >> >>> master status page
> >> >>>
> >> >>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
> >> >>>
> >> >>> >We ran the command unassign 'REGIONNAME',true the output is
> >> >>>"completed in
> >> >>> >57 sec", but still region is in transition.
> >> >>> >
> >> >>> >
> >> >>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
> >> >>> ><sa...@gmail.com>wrote:
> >> >>> >
> >> >>> >> Was the unassigning successful ?
> >> >>> >>
> >> >>> >> If not you can force it by - unassign 'REGIONNAME', true
> >> >>> >>
> >> >>> >> Regards,
> >> >>> >> Samar
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com>
> wrote:
> >> >>> >>
> >> >>> >> >i issued unassign and close_region both but the region is still
> in
> >> >>> >> >transition also I deleted the .META entry for the region. Do I
> >> >>>need to
> >> >>> >> >restart master ?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> >> >>> >> ><sa...@gmail.com>wrote:
> >> >>> >> >
> >> >>> >> >> Regions should never overlap
> >> >>> >> >> Incase a region is in transition for a long long time, you
> could
> >> >>> >> >>possibly
> >> >>> >> >> force Unassign a region , if you are ok to lose the region.
> >> >>> >> >> Regards,
> >> >>> >> >> Samar
> >> >>> >> >>
> >> >>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com>
> >> >>>wrote:
> >> >>> >> >>
> >> >>> >> >> > are always in transition ? Because of the
> >> >>> >> >> >cluster is not balancing and regionservers are going down one
> >> by
> >> >>> >>one.
> >> >>> >> >>As
> >> >>> >> >> >only some regions servers are handling the requests.
> >> >>> >> >> >
> >> >>> >> >> >Another wierd thing is the region stuck in transition
> startkey
> >> >>> >>matches
> >> >>> >> >> >with
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >>
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >--
> >> >>> >> >Thank you
> >> >>> >> >Kiran Sarvabhotla
> >> >>> >> >
> >> >>> >> >-----Even a correct decision is wrong when it is taken late
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >--
> >> >>> >Thank you
> >> >>> >Kiran Sarvabhotla
> >> >>> >
> >> >>> >-----Even a correct decision is wrong when it is taken late
> >> >>>
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> Thank you
> >> >> Kiran Sarvabhotla
> >> >>
> >> >> -----Even a correct decision is wrong when it is taken late
> >>
> >>
> >>
> >
> >
> > --
> > Thank you
> > Kiran Sarvabhotla
> >
> > -----Even a correct decision is wrong when it is taken late
> >
> >
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
>
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
after some searching i found that main reason for regions with same keys is
splitting and it is the culprit in our case for inconsistency. But I set my
filesize to a very large size but i am unsure why still splitting is
happening
On Fri, Feb 8, 2013 at 1:20 PM, kiran <ki...@gmail.com> wrote:
> Thanks samar and ramakrishna
>
> I did use option 2 and deleted /hbase from the zkcli shell. I restarted
> the cluster now there are more regions in transition. Will it take some
> time to complete since it is a new assignment ?
>
> My hase version is 0.94.1, and I have a doubt with the property
> hbase.hregion.max.filesize. I set it to a very large value of 100GB. But
> some how table regions are not splitted as per the property.
>
>
> On Fri, Feb 8, 2013 at 12:45 PM, samar kumar <sa...@gmail.com>wrote:
>
>> May be ./bin/hbase hbck -repairHoles could be helpful..
>>
>>
>> On 08/02/13 12:38 PM, "Samir Ahmic" <ah...@gmail.com> wrote:
>>
>> >Hi, Kiran
>> >
>> >Welcome to beautiful world of HBase transition states :) . When i face
>> >RIT issue this are steps that i use to resolve it:
>> >
>> >1. hbase hbck -fixAssignments (this depends on your version of hbase
>> >it can be also just -fix)
>> >
>> >If you don't have luck with 1. then you will need manual intervention:
>> >Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
>> >(or delete ) /hbase, and then restart cluster.
>> >
>> >This should help you resolve RITs.
>> >
>> >Regards
>> >
>> >
>> >
>> >On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com>
>> wrote:
>> >> PENDING_OPEN
>> >>
>> >>
>> >> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar
>> >><sa...@gmail.com>wrote:
>> >>
>> >>> Can you mention the state of the region.. You can find the details in
>> >>>you
>> >>> master status page
>> >>>
>> >>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
>> >>>
>> >>> >We ran the command unassign 'REGIONNAME',true the output is
>> >>>"completed in
>> >>> >57 sec", but still region is in transition.
>> >>> >
>> >>> >
>> >>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
>> >>> ><sa...@gmail.com>wrote:
>> >>> >
>> >>> >> Was the unassigning successful ?
>> >>> >>
>> >>> >> If not you can force it by - unassign 'REGIONNAME', true
>> >>> >>
>> >>> >> Regards,
>> >>> >> Samar
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>> >>> >>
>> >>> >> >i issued unassign and close_region both but the region is still in
>> >>> >> >transition also I deleted the .META entry for the region. Do I
>> >>>need to
>> >>> >> >restart master ?
>> >>> >> >
>> >>> >> >
>> >>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
>> >>> >> ><sa...@gmail.com>wrote:
>> >>> >> >
>> >>> >> >> Regions should never overlap
>> >>> >> >> Incase a region is in transition for a long long time, you could
>> >>> >> >>possibly
>> >>> >> >> force Unassign a region , if you are ok to lose the region.
>> >>> >> >> Regards,
>> >>> >> >> Samar
>> >>> >> >>
>> >>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com>
>> >>>wrote:
>> >>> >> >>
>> >>> >> >> > are always in transition ? Because of the
>> >>> >> >> >cluster is not balancing and regionservers are going down one
>> by
>> >>> >>one.
>> >>> >> >>As
>> >>> >> >> >only some regions servers are handling the requests.
>> >>> >> >> >
>> >>> >> >> >Another wierd thing is the region stuck in transition startkey
>> >>> >>matches
>> >>> >> >> >with
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >
>> >>> >> >
>> >>> >> >--
>> >>> >> >Thank you
>> >>> >> >Kiran Sarvabhotla
>> >>> >> >
>> >>> >> >-----Even a correct decision is wrong when it is taken late
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >
>> >>> >
>> >>> >--
>> >>> >Thank you
>> >>> >Kiran Sarvabhotla
>> >>> >
>> >>> >-----Even a correct decision is wrong when it is taken late
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Thank you
>> >> Kiran Sarvabhotla
>> >>
>> >> -----Even a correct decision is wrong when it is taken late
>>
>>
>>
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
Thanks samar and ramakrishna
I did use option 2 and deleted /hbase from the zkcli shell. I restarted the
cluster now there are more regions in transition. Will it take some time to
complete since it is a new assignment ?
My hase version is 0.94.1, and I have a doubt with the property
hbase.hregion.max.filesize. I set it to a very large value of 100GB. But
some how table regions are not splitted as per the property.
On Fri, Feb 8, 2013 at 12:45 PM, samar kumar <sa...@gmail.com>wrote:
> May be ./bin/hbase hbck -repairHoles could be helpful..
>
>
> On 08/02/13 12:38 PM, "Samir Ahmic" <ah...@gmail.com> wrote:
>
> >Hi, Kiran
> >
> >Welcome to beautiful world of HBase transition states :) . When i face
> >RIT issue this are steps that i use to resolve it:
> >
> >1. hbase hbck -fixAssignments (this depends on your version of hbase
> >it can be also just -fix)
> >
> >If you don't have luck with 1. then you will need manual intervention:
> >Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
> >(or delete ) /hbase, and then restart cluster.
> >
> >This should help you resolve RITs.
> >
> >Regards
> >
> >
> >
> >On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com>
> wrote:
> >> PENDING_OPEN
> >>
> >>
> >> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar
> >><sa...@gmail.com>wrote:
> >>
> >>> Can you mention the state of the region.. You can find the details in
> >>>you
> >>> master status page
> >>>
> >>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
> >>>
> >>> >We ran the command unassign 'REGIONNAME',true the output is
> >>>"completed in
> >>> >57 sec", but still region is in transition.
> >>> >
> >>> >
> >>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
> >>> ><sa...@gmail.com>wrote:
> >>> >
> >>> >> Was the unassigning successful ?
> >>> >>
> >>> >> If not you can force it by - unassign 'REGIONNAME', true
> >>> >>
> >>> >> Regards,
> >>> >> Samar
> >>> >>
> >>> >>
> >>> >>
> >>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
> >>> >>
> >>> >> >i issued unassign and close_region both but the region is still in
> >>> >> >transition also I deleted the .META entry for the region. Do I
> >>>need to
> >>> >> >restart master ?
> >>> >> >
> >>> >> >
> >>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> >>> >> ><sa...@gmail.com>wrote:
> >>> >> >
> >>> >> >> Regions should never overlap
> >>> >> >> Incase a region is in transition for a long long time, you could
> >>> >> >>possibly
> >>> >> >> force Unassign a region , if you are ok to lose the region.
> >>> >> >> Regards,
> >>> >> >> Samar
> >>> >> >>
> >>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com>
> >>>wrote:
> >>> >> >>
> >>> >> >> > are always in transition ? Because of the
> >>> >> >> >cluster is not balancing and regionservers are going down one by
> >>> >>one.
> >>> >> >>As
> >>> >> >> >only some regions servers are handling the requests.
> >>> >> >> >
> >>> >> >> >Another wierd thing is the region stuck in transition startkey
> >>> >>matches
> >>> >> >> >with
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >> >--
> >>> >> >Thank you
> >>> >> >Kiran Sarvabhotla
> >>> >> >
> >>> >> >-----Even a correct decision is wrong when it is taken late
> >>> >>
> >>> >>
> >>> >>
> >>> >
> >>> >
> >>> >--
> >>> >Thank you
> >>> >Kiran Sarvabhotla
> >>> >
> >>> >-----Even a correct decision is wrong when it is taken late
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Thank you
> >> Kiran Sarvabhotla
> >>
> >> -----Even a correct decision is wrong when it is taken late
>
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Yes, RIT states were bit confusing for the end user. In the latest version
lot of efforts have gone into solve this problems. Hope for the best in the
future :)
Regards
Ram
On Fri, Feb 8, 2013 at 12:38 PM, Samir Ahmic <ah...@gmail.com> wrote:
> Hi, Kiran
>
> Welcome to beautiful world of HBase transition states :) . When i face
> RIT issue this are steps that i use to resolve it:
>
> 1. hbase hbck -fixAssignments (this depends on your version of hbase
> it can be also just -fix)
>
> If you don't have luck with 1. then you will need manual intervention:
> Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
> (or delete ) /hbase, and then restart cluster.
>
> This should help you resolve RITs.
>
> Regards
>
>
>
> On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com> wrote:
> > PENDING_OPEN
> >
> >
> > On Fri, Feb 8, 2013 at 12:16 PM, samar kumar <samar.opensource@gmail.com
> >wrote:
> >
> >> Can you mention the state of the region.. You can find the details in
> you
> >> master status page
> >>
> >> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
> >>
> >> >We ran the command unassign 'REGIONNAME',true the output is "completed
> in
> >> >57 sec", but still region is in transition.
> >> >
> >> >
> >> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
> >> ><sa...@gmail.com>wrote:
> >> >
> >> >> Was the unassigning successful ?
> >> >>
> >> >> If not you can force it by - unassign 'REGIONNAME', true
> >> >>
> >> >> Regards,
> >> >> Samar
> >> >>
> >> >>
> >> >>
> >> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
> >> >>
> >> >> >i issued unassign and close_region both but the region is still in
> >> >> >transition also I deleted the .META entry for the region. Do I need
> to
> >> >> >restart master ?
> >> >> >
> >> >> >
> >> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> >> >> ><sa...@gmail.com>wrote:
> >> >> >
> >> >> >> Regions should never overlap
> >> >> >> Incase a region is in transition for a long long time, you could
> >> >> >>possibly
> >> >> >> force Unassign a region , if you are ok to lose the region.
> >> >> >> Regards,
> >> >> >> Samar
> >> >> >>
> >> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com>
> wrote:
> >> >> >>
> >> >> >> > are always in transition ? Because of the
> >> >> >> >cluster is not balancing and regionservers are going down one by
> >> >>one.
> >> >> >>As
> >> >> >> >only some regions servers are handling the requests.
> >> >> >> >
> >> >> >> >Another wierd thing is the region stuck in transition startkey
> >> >>matches
> >> >> >> >with
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >--
> >> >> >Thank you
> >> >> >Kiran Sarvabhotla
> >> >> >
> >> >> >-----Even a correct decision is wrong when it is taken late
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >> >--
> >> >Thank you
> >> >Kiran Sarvabhotla
> >> >
> >> >-----Even a correct decision is wrong when it is taken late
> >>
> >>
> >>
> >
> >
> > --
> > Thank you
> > Kiran Sarvabhotla
> >
> > -----Even a correct decision is wrong when it is taken late
>
Re: Regions in transition
Posted by samar kumar <sa...@gmail.com>.
May be ./bin/hbase hbck -repairHoles could be helpful..
On 08/02/13 12:38 PM, "Samir Ahmic" <ah...@gmail.com> wrote:
>Hi, Kiran
>
>Welcome to beautiful world of HBase transition states :) . When i face
>RIT issue this are steps that i use to resolve it:
>
>1. hbase hbck -fixAssignments (this depends on your version of hbase
>it can be also just -fix)
>
>If you don't have luck with 1. then you will need manual intervention:
>Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
>(or delete ) /hbase, and then restart cluster.
>
>This should help you resolve RITs.
>
>Regards
>
>
>
>On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com> wrote:
>> PENDING_OPEN
>>
>>
>> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar
>><sa...@gmail.com>wrote:
>>
>>> Can you mention the state of the region.. You can find the details in
>>>you
>>> master status page
>>>
>>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
>>>
>>> >We ran the command unassign 'REGIONNAME',true the output is
>>>"completed in
>>> >57 sec", but still region is in transition.
>>> >
>>> >
>>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
>>> ><sa...@gmail.com>wrote:
>>> >
>>> >> Was the unassigning successful ?
>>> >>
>>> >> If not you can force it by - unassign 'REGIONNAME', true
>>> >>
>>> >> Regards,
>>> >> Samar
>>> >>
>>> >>
>>> >>
>>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>>> >>
>>> >> >i issued unassign and close_region both but the region is still in
>>> >> >transition also I deleted the .META entry for the region. Do I
>>>need to
>>> >> >restart master ?
>>> >> >
>>> >> >
>>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
>>> >> ><sa...@gmail.com>wrote:
>>> >> >
>>> >> >> Regions should never overlap
>>> >> >> Incase a region is in transition for a long long time, you could
>>> >> >>possibly
>>> >> >> force Unassign a region , if you are ok to lose the region.
>>> >> >> Regards,
>>> >> >> Samar
>>> >> >>
>>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com>
>>>wrote:
>>> >> >>
>>> >> >> > are always in transition ? Because of the
>>> >> >> >cluster is not balancing and regionservers are going down one by
>>> >>one.
>>> >> >>As
>>> >> >> >only some regions servers are handling the requests.
>>> >> >> >
>>> >> >> >Another wierd thing is the region stuck in transition startkey
>>> >>matches
>>> >> >> >with
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >
>>> >> >
>>> >> >--
>>> >> >Thank you
>>> >> >Kiran Sarvabhotla
>>> >> >
>>> >> >-----Even a correct decision is wrong when it is taken late
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> >--
>>> >Thank you
>>> >Kiran Sarvabhotla
>>> >
>>> >-----Even a correct decision is wrong when it is taken late
>>>
>>>
>>>
>>
>>
>> --
>> Thank you
>> Kiran Sarvabhotla
>>
>> -----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by Samir Ahmic <ah...@gmail.com>.
Hi, Kiran
Welcome to beautiful world of HBase transition states :) . When i face
RIT issue this are steps that i use to resolve it:
1. hbase hbck -fixAssignments (this depends on your version of hbase
it can be also just -fix)
If you don't have luck with 1. then you will need manual intervention:
Remove hbase znodes from zookeeper (use hbase zkcli) and options rmr
(or delete ) /hbase, and then restart cluster.
This should help you resolve RITs.
Regards
On Fri, Feb 8, 2013 at 7:50 AM, kiran <ki...@gmail.com> wrote:
> PENDING_OPEN
>
>
> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar <sa...@gmail.com>wrote:
>
>> Can you mention the state of the region.. You can find the details in you
>> master status page
>>
>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
>>
>> >We ran the command unassign 'REGIONNAME',true the output is "completed in
>> >57 sec", but still region is in transition.
>> >
>> >
>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
>> ><sa...@gmail.com>wrote:
>> >
>> >> Was the unassigning successful ?
>> >>
>> >> If not you can force it by - unassign 'REGIONNAME', true
>> >>
>> >> Regards,
>> >> Samar
>> >>
>> >>
>> >>
>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>> >>
>> >> >i issued unassign and close_region both but the region is still in
>> >> >transition also I deleted the .META entry for the region. Do I need to
>> >> >restart master ?
>> >> >
>> >> >
>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
>> >> ><sa...@gmail.com>wrote:
>> >> >
>> >> >> Regions should never overlap
>> >> >> Incase a region is in transition for a long long time, you could
>> >> >>possibly
>> >> >> force Unassign a region , if you are ok to lose the region.
>> >> >> Regards,
>> >> >> Samar
>> >> >>
>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
>> >> >>
>> >> >> > are always in transition ? Because of the
>> >> >> >cluster is not balancing and regionservers are going down one by
>> >>one.
>> >> >>As
>> >> >> >only some regions servers are handling the requests.
>> >> >> >
>> >> >> >Another wierd thing is the region stuck in transition startkey
>> >>matches
>> >> >> >with
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >--
>> >> >Thank you
>> >> >Kiran Sarvabhotla
>> >> >
>> >> >-----Even a correct decision is wrong when it is taken late
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >Thank you
>> >Kiran Sarvabhotla
>> >
>> >-----Even a correct decision is wrong when it is taken late
>>
>>
>>
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
Hi,
I tried to run hbase hbck -fixMeta after master restart. and in the output
I got
There is a hole in the region chain between <regionname> You need to create
a new .regioninfo and region dir in hdfs to plug the hole for about 10 odd
regions
What does this mean ?
On Fri, Feb 8, 2013 at 12:20 PM, kiran <ki...@gmail.com> wrote:
> PENDING_OPEN
>
>
> On Fri, Feb 8, 2013 at 12:16 PM, samar kumar <sa...@gmail.com>wrote:
>
>> Can you mention the state of the region.. You can find the details in you
>> master status page
>>
>> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
>>
>> >We ran the command unassign 'REGIONNAME',true the output is "completed in
>> >57 sec", but still region is in transition.
>> >
>> >
>> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
>> ><sa...@gmail.com>wrote:
>> >
>> >> Was the unassigning successful ?
>> >>
>> >> If not you can force it by - unassign 'REGIONNAME', true
>> >>
>> >> Regards,
>> >> Samar
>> >>
>> >>
>> >>
>> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>> >>
>> >> >i issued unassign and close_region both but the region is still in
>> >> >transition also I deleted the .META entry for the region. Do I need to
>> >> >restart master ?
>> >> >
>> >> >
>> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
>> >> ><sa...@gmail.com>wrote:
>> >> >
>> >> >> Regions should never overlap
>> >> >> Incase a region is in transition for a long long time, you could
>> >> >>possibly
>> >> >> force Unassign a region , if you are ok to lose the region.
>> >> >> Regards,
>> >> >> Samar
>> >> >>
>> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
>> >> >>
>> >> >> > are always in transition ? Because of the
>> >> >> >cluster is not balancing and regionservers are going down one by
>> >>one.
>> >> >>As
>> >> >> >only some regions servers are handling the requests.
>> >> >> >
>> >> >> >Another wierd thing is the region stuck in transition startkey
>> >>matches
>> >> >> >with
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >--
>> >> >Thank you
>> >> >Kiran Sarvabhotla
>> >> >
>> >> >-----Even a correct decision is wrong when it is taken late
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >Thank you
>> >Kiran Sarvabhotla
>> >
>> >-----Even a correct decision is wrong when it is taken late
>>
>>
>>
>
>
> --
> Thank you
> Kiran Sarvabhotla
>
> -----Even a correct decision is wrong when it is taken late
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
PENDING_OPEN
On Fri, Feb 8, 2013 at 12:16 PM, samar kumar <sa...@gmail.com>wrote:
> Can you mention the state of the region.. You can find the details in you
> master status page
>
> On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
>
> >We ran the command unassign 'REGIONNAME',true the output is "completed in
> >57 sec", but still region is in transition.
> >
> >
> >On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
> ><sa...@gmail.com>wrote:
> >
> >> Was the unassigning successful ?
> >>
> >> If not you can force it by - unassign 'REGIONNAME', true
> >>
> >> Regards,
> >> Samar
> >>
> >>
> >>
> >> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
> >>
> >> >i issued unassign and close_region both but the region is still in
> >> >transition also I deleted the .META entry for the region. Do I need to
> >> >restart master ?
> >> >
> >> >
> >> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> >> ><sa...@gmail.com>wrote:
> >> >
> >> >> Regions should never overlap
> >> >> Incase a region is in transition for a long long time, you could
> >> >>possibly
> >> >> force Unassign a region , if you are ok to lose the region.
> >> >> Regards,
> >> >> Samar
> >> >>
> >> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
> >> >>
> >> >> > are always in transition ? Because of the
> >> >> >cluster is not balancing and regionservers are going down one by
> >>one.
> >> >>As
> >> >> >only some regions servers are handling the requests.
> >> >> >
> >> >> >Another wierd thing is the region stuck in transition startkey
> >>matches
> >> >> >with
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >> >--
> >> >Thank you
> >> >Kiran Sarvabhotla
> >> >
> >> >-----Even a correct decision is wrong when it is taken late
> >>
> >>
> >>
> >
> >
> >--
> >Thank you
> >Kiran Sarvabhotla
> >
> >-----Even a correct decision is wrong when it is taken late
>
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by samar kumar <sa...@gmail.com>.
Can you mention the state of the region.. You can find the details in you
master status page
On 08/02/13 12:09 PM, "kiran" <ki...@gmail.com> wrote:
>We ran the command unassign 'REGIONNAME',true the output is "completed in
>57 sec", but still region is in transition.
>
>
>On Fri, Feb 8, 2013 at 12:05 PM, samar kumar
><sa...@gmail.com>wrote:
>
>> Was the unassigning successful ?
>>
>> If not you can force it by - unassign 'REGIONNAME', true
>>
>> Regards,
>> Samar
>>
>>
>>
>> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>>
>> >i issued unassign and close_region both but the region is still in
>> >transition also I deleted the .META entry for the region. Do I need to
>> >restart master ?
>> >
>> >
>> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
>> ><sa...@gmail.com>wrote:
>> >
>> >> Regions should never overlap
>> >> Incase a region is in transition for a long long time, you could
>> >>possibly
>> >> force Unassign a region , if you are ok to lose the region.
>> >> Regards,
>> >> Samar
>> >>
>> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
>> >>
>> >> > are always in transition ? Because of the
>> >> >cluster is not balancing and regionservers are going down one by
>>one.
>> >>As
>> >> >only some regions servers are handling the requests.
>> >> >
>> >> >Another wierd thing is the region stuck in transition startkey
>>matches
>> >> >with
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >Thank you
>> >Kiran Sarvabhotla
>> >
>> >-----Even a correct decision is wrong when it is taken late
>>
>>
>>
>
>
>--
>Thank you
>Kiran Sarvabhotla
>
>-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
We ran the command unassign 'REGIONNAME',true the output is "completed in
57 sec", but still region is in transition.
On Fri, Feb 8, 2013 at 12:05 PM, samar kumar <sa...@gmail.com>wrote:
> Was the unassigning successful ?
>
> If not you can force it by - unassign 'REGIONNAME', true
>
> Regards,
> Samar
>
>
>
> On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>
> >i issued unassign and close_region both but the region is still in
> >transition also I deleted the .META entry for the region. Do I need to
> >restart master ?
> >
> >
> >On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
> ><sa...@gmail.com>wrote:
> >
> >> Regions should never overlap
> >> Incase a region is in transition for a long long time, you could
> >>possibly
> >> force Unassign a region , if you are ok to lose the region.
> >> Regards,
> >> Samar
> >>
> >> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
> >>
> >> > are always in transition ? Because of the
> >> >cluster is not balancing and regionservers are going down one by one.
> >>As
> >> >only some regions servers are handling the requests.
> >> >
> >> >Another wierd thing is the region stuck in transition startkey matches
> >> >with
> >>
> >>
> >>
> >
> >
> >--
> >Thank you
> >Kiran Sarvabhotla
> >
> >-----Even a correct decision is wrong when it is taken late
>
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by samar kumar <sa...@gmail.com>.
Was the unassigning successful ?
If not you can force it by - unassign 'REGIONNAME', true
Regards,
Samar
On 08/02/13 12:01 PM, "kiran" <ki...@gmail.com> wrote:
>i issued unassign and close_region both but the region is still in
>transition also I deleted the .META entry for the region. Do I need to
>restart master ?
>
>
>On Fri, Feb 8, 2013 at 11:18 AM, samar kumar
><sa...@gmail.com>wrote:
>
>> Regions should never overlap
>> Incase a region is in transition for a long long time, you could
>>possibly
>> force Unassign a region , if you are ok to lose the region.
>> Regards,
>> Samar
>>
>> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
>>
>> > are always in transition ? Because of the
>> >cluster is not balancing and regionservers are going down one by one.
>>As
>> >only some regions servers are handling the requests.
>> >
>> >Another wierd thing is the region stuck in transition startkey matches
>> >with
>>
>>
>>
>
>
>--
>Thank you
>Kiran Sarvabhotla
>
>-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by kiran <ki...@gmail.com>.
i issued unassign and close_region both but the region is still in
transition also I deleted the .META entry for the region. Do I need to
restart master ?
On Fri, Feb 8, 2013 at 11:18 AM, samar kumar <sa...@gmail.com>wrote:
> Regions should never overlap
> Incase a region is in transition for a long long time, you could possibly
> force Unassign a region , if you are ok to lose the region.
> Regards,
> Samar
>
> On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
>
> > are always in transition ? Because of the
> >cluster is not balancing and regionservers are going down one by one. As
> >only some regions servers are handling the requests.
> >
> >Another wierd thing is the region stuck in transition startkey matches
> >with
>
>
>
--
Thank you
Kiran Sarvabhotla
-----Even a correct decision is wrong when it is taken late
Re: Regions in transition
Posted by samar kumar <sa...@gmail.com>.
Regions should never overlap
Incase a region is in transition for a long long time, you could possibly
force Unassign a region , if you are ok to lose the region.
Regards,
Samar
On 08/02/13 11:14 AM, "kiran" <ki...@gmail.com> wrote:
> are always in transition ? Because of the
>cluster is not balancing and regionservers are going down one by one. As
>only some regions servers are handling the requests.
>
>Another wierd thing is the region stuck in transition startkey matches
>with