You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Lukáš Drbal <lu...@gmail.com> on 2012/10/15 14:38:31 UTC
Region in Transition
Hi all,
i have now a big problem with one region. This region is allways in
transition and i don't no how fix it.
I run "hbase hbck -repair" and this ended with:
INFO util.HBaseFsckRepair: Region still in transition, waiting for it
to become assigned: {NAME =>
'twitter_tweets,08000e806a7b8ba7d6cdd5c9c0956e7e,1350294699032.3a932a1e2b15abbf7539ca8ad761ecfc.',
STARTKEY => '08000e806a7b8ba7d6cdd5c9c0956e7e', ENDKEY =>
'1002e24a61f8ea1ce10267e09e377aa5', ENCODED =>
3a932a1e2b15abbf7539ca8ad761ecfc,
How i can bring it back online?
Re: Region in Transition
Posted by Bryan Beaudreault <bb...@hubspot.com>.
In the past when I have seen regions get locked in transition it is usually
a problem with the HMaster. Seemingly a transition starts and succeeds
between region servers, but the HMaster may miss part of that communication
and think it is still in transition. Then it keeps retrying but the node
is not where it expects anymore. This doesn't affect normal operations,
except that the balancer cannot run while something is in transition.
Restarting the HMaster always fixed this for me and had no adverse affects
that I could see.
On Mon, Oct 15, 2012 at 10:28 AM, Kevin O'dell <ke...@cloudera.com>wrote:
> Lukas,
>
> Sure, how long has it been since the restart? You will need to give the
> regions time to transition and logs time to split. Did the region
> in transition properly? Can you please out a pastebin together of hbck
> -details full output so that I can take a look at it. Once I have reviewed
> it, I should be able to provide you with a plan to out this back together.
>
> On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com>
> wrote:
>
> > Hello Kevin,
> >
> > thanks for response. I clear it now and start cluster, but now i have
> > many other's regions :(
> > Here is output from hbase hbck -fixmeta
> > http://pastebin.com/HqsPVLMi
> >
> > Any next hint?
> >
> > Thanks a lot
> >
> > 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> > > Have you tried clearing out your Znode information? Typically, when I
> > have
> > > encountered a RIT, we will bring down HBase and go to the ZKcli and
> clear
> > > out /hbase. What do you see in the logs pertaining to the region? If
> it
> > > is a region that has bad hfiles or something like that you will not be
> > able
> > > to easily get rid of it through clearing out the ZK.
> > >
> > > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com>
> > wrote:
> > >
> > >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> > >>
> > >> I found this in master log http://pastebin.com/CYt5PZCL and this line
> > >> are repeated for all region servers.
> > >>
> > >> Can someone help me please?
> > >>
> > >
> > >
> > >
> > > --
> > > Kevin O'Dell
> > > Customer Operations Engineer, Cloudera
> >
> >
> >
> > --
> > Save The World - http://www.worldcommunitygrid.org/
> > http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >
> > LesTR
> >
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>
Re: Region in Transition
Posted by Lukáš Drbal <lu...@gmail.com>.
Sorry Kevin, english isn't my primary language.Iam so sorry.
What i need:
- i have a lot of data actualy in /hbase.old
- new data are saved to /hbase
and now i need "combine" all this data, but exists rows from /hbase
can't be rewrited by rows from /hbase.old.
Maybe i can do it like this:
hadoop fs -mv /hbase /hbase.old - this replace all exists old rows?
After this i can move /hbase.old to /hbase and run hbck.
Both with stopped cluster.
Lukas
2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> Lukas,
>
> I am not sure I understand what you are saying there. What I thought you
> did was:
>
> hadoop fs -mv /hbase /tmp/hbase.old
>
> Restart HBase
>
> You should now have a clean HBase
>
> hadoop fs -mv /tmp/hbase.old/<table_name>
>
> Once you have moved all of the tables
>
> ./bin/hbase hbck -fixMeta -fixAssignments
>
> This should put your hbase back into a happy state, if it doesn't not you
> had underlying issues that need resolving.
>
> On Mon, Oct 15, 2012 at 12:09 PM, Lukáš Drbal <lu...@gmail.com> wrote:
>
>> Wow, its perfect "hack" but what about "exists" rows?
>> For example when i have saved new version from row in /hbase i can't
>> replace it from /hbase.old
>>
>> Thanks for time
>>
>> Lukas
>>
>> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> > Lukas,
>> >
>> > A little trick you can use is to just copy the table directories into
>> > your new /hbase dir and then use:
>> >
>> > hbck -fixMeta -fixAssignments
>> >
>> > This will pull in the tables into your new empty META. If once you do
>> this
>> > Regions get stuck in transition again, you have bigger problems.
>> >
>> > On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <lu...@gmail.com>
>> wrote:
>> >
>> >> Hi again.
>> >>
>> >> > Does FSCK come back clean? Are those the regions showing in
>> transition?
>> >> > We are not going to be able to get a clear idea of what to do next
>> until
>> >> > we gather some more data. At this point running repairs could put
>> your
>> >> > data in jeopardy.
>> >>
>> >>
>> >> Ye, this is in transition, but from hadoop fsck are all ok and
>> >> consistent - reported only 14 undereplicated blocks from another
>> >> directory (not /hbase).
>> >>
>> >> I must bring cluster back online, so i move /hbase on hdfs to
>> >> /hbase.old and start clear cluster with new tables :(
>> >>
>> >> Can you give me please any suggestion how i can put data from old
>> >> files to new tables?
>> >>
>> >> Thanks
>> >>
>> >> My today's night will be very long :(
>> >>
>> >> Lukas
>> >>
>> >> >
>> >> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <lu...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Kevin,
>> >> >>
>> >> >> Its more than one hour and region are still in transition.
>> >> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>> >> >>
>> >> >> Thanks for time
>> >> >>
>> >> >> Lukas
>> >> >>
>> >> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> >> >> > Lukas,
>> >> >> >
>> >> >> > Sure, how long has it been since the restart? You will need to
>> give
>> >> >> the
>> >> >> > regions time to transition and logs time to split. Did the region
>> >> >> > in transition properly? Can you please out a pastebin together of
>> >> hbck
>> >> >> > -details full output so that I can take a look at it. Once I have
>> >> >> reviewed
>> >> >> > it, I should be able to provide you with a plan to out this back
>> >> >> together.
>> >> >> >
>> >> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <
>> lukas.drbal@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Hello Kevin,
>> >> >> >>
>> >> >> >> thanks for response. I clear it now and start cluster, but now i
>> have
>> >> >> >> many other's regions :(
>> >> >> >> Here is output from hbase hbck -fixmeta
>> >> >> >> http://pastebin.com/HqsPVLMi
>> >> >> >>
>> >> >> >> Any next hint?
>> >> >> >>
>> >> >> >> Thanks a lot
>> >> >> >>
>> >> >> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> >> >> >> > Have you tried clearing out your Znode information? Typically,
>> >> when I
>> >> >> >> have
>> >> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli
>> and
>> >> >> clear
>> >> >> >> > out /hbase. What do you see in the logs pertaining to the
>> region?
>> >> >> If it
>> >> >> >> > is a region that has bad hfiles or something like that you will
>> >> not be
>> >> >> >> able
>> >> >> >> > to easily get rid of it through clearing out the ZK.
>> >> >> >> >
>> >> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
>> >> lukas.drbal@gmail.com>
>> >> >> >> wrote:
>> >> >> >> >
>> >> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >> >> >> >>
>> >> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and
>> this
>> >> >> line
>> >> >> >> >> are repeated for all region servers.
>> >> >> >> >>
>> >> >> >> >> Can someone help me please?
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > Kevin O'Dell
>> >> >> >> > Customer Operations Engineer, Cloudera
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Save The World - http://www.worldcommunitygrid.org/
>> >> >> >>
>> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >> >> >>
>> >> >> >> LesTR
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Kevin O'Dell
>> >> >> > Customer Operations Engineer, Cloudera
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Save The World - http://www.worldcommunitygrid.org/
>> >> >>
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >> >>
>> >> >> LesTR
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kevin O'Dell
>> >> > Customer Operations Engineer, Cloudera
>> >>
>> >>
>> >>
>> >> --
>> >> Save The World - http://www.worldcommunitygrid.org/
>> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >>
>> >> LesTR
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>>
>>
>> --
>> Save The World - http://www.worldcommunitygrid.org/
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>>
>> LesTR
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
LesTR
Re: Region in Transition
Posted by Kevin O'dell <ke...@cloudera.com>.
Lukas,
I am not sure I understand what you are saying there. What I thought you
did was:
hadoop fs -mv /hbase /tmp/hbase.old
Restart HBase
You should now have a clean HBase
hadoop fs -mv /tmp/hbase.old/<table_name>
Once you have moved all of the tables
./bin/hbase hbck -fixMeta -fixAssignments
This should put your hbase back into a happy state, if it doesn't not you
had underlying issues that need resolving.
On Mon, Oct 15, 2012 at 12:09 PM, Lukáš Drbal <lu...@gmail.com> wrote:
> Wow, its perfect "hack" but what about "exists" rows?
> For example when i have saved new version from row in /hbase i can't
> replace it from /hbase.old
>
> Thanks for time
>
> Lukas
>
> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> > Lukas,
> >
> > A little trick you can use is to just copy the table directories into
> > your new /hbase dir and then use:
> >
> > hbck -fixMeta -fixAssignments
> >
> > This will pull in the tables into your new empty META. If once you do
> this
> > Regions get stuck in transition again, you have bigger problems.
> >
> > On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <lu...@gmail.com>
> wrote:
> >
> >> Hi again.
> >>
> >> > Does FSCK come back clean? Are those the regions showing in
> transition?
> >> > We are not going to be able to get a clear idea of what to do next
> until
> >> > we gather some more data. At this point running repairs could put
> your
> >> > data in jeopardy.
> >>
> >>
> >> Ye, this is in transition, but from hadoop fsck are all ok and
> >> consistent - reported only 14 undereplicated blocks from another
> >> directory (not /hbase).
> >>
> >> I must bring cluster back online, so i move /hbase on hdfs to
> >> /hbase.old and start clear cluster with new tables :(
> >>
> >> Can you give me please any suggestion how i can put data from old
> >> files to new tables?
> >>
> >> Thanks
> >>
> >> My today's night will be very long :(
> >>
> >> Lukas
> >>
> >> >
> >> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <lu...@gmail.com>
> >> wrote:
> >> >
> >> >> Kevin,
> >> >>
> >> >> Its more than one hour and region are still in transition.
> >> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
> >> >>
> >> >> Thanks for time
> >> >>
> >> >> Lukas
> >> >>
> >> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> >> >> > Lukas,
> >> >> >
> >> >> > Sure, how long has it been since the restart? You will need to
> give
> >> >> the
> >> >> > regions time to transition and logs time to split. Did the region
> >> >> > in transition properly? Can you please out a pastebin together of
> >> hbck
> >> >> > -details full output so that I can take a look at it. Once I have
> >> >> reviewed
> >> >> > it, I should be able to provide you with a plan to out this back
> >> >> together.
> >> >> >
> >> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <
> lukas.drbal@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hello Kevin,
> >> >> >>
> >> >> >> thanks for response. I clear it now and start cluster, but now i
> have
> >> >> >> many other's regions :(
> >> >> >> Here is output from hbase hbck -fixmeta
> >> >> >> http://pastebin.com/HqsPVLMi
> >> >> >>
> >> >> >> Any next hint?
> >> >> >>
> >> >> >> Thanks a lot
> >> >> >>
> >> >> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> >> >> >> > Have you tried clearing out your Znode information? Typically,
> >> when I
> >> >> >> have
> >> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli
> and
> >> >> clear
> >> >> >> > out /hbase. What do you see in the logs pertaining to the
> region?
> >> >> If it
> >> >> >> > is a region that has bad hfiles or something like that you will
> >> not be
> >> >> >> able
> >> >> >> > to easily get rid of it through clearing out the ZK.
> >> >> >> >
> >> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
> >> lukas.drbal@gmail.com>
> >> >> >> wrote:
> >> >> >> >
> >> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >> >> >> >>
> >> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and
> this
> >> >> line
> >> >> >> >> are repeated for all region servers.
> >> >> >> >>
> >> >> >> >> Can someone help me please?
> >> >> >> >>
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > Kevin O'Dell
> >> >> >> > Customer Operations Engineer, Cloudera
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Save The World - http://www.worldcommunitygrid.org/
> >> >> >>
> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >> >> >>
> >> >> >> LesTR
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Kevin O'Dell
> >> >> > Customer Operations Engineer, Cloudera
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Save The World - http://www.worldcommunitygrid.org/
> >> >>
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >> >>
> >> >> LesTR
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Kevin O'Dell
> >> > Customer Operations Engineer, Cloudera
> >>
> >>
> >>
> >> --
> >> Save The World - http://www.worldcommunitygrid.org/
> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >>
> >> LesTR
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
>
> --
> Save The World - http://www.worldcommunitygrid.org/
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>
> LesTR
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: Region in Transition
Posted by Lukáš Drbal <lu...@gmail.com>.
Wow, its perfect "hack" but what about "exists" rows?
For example when i have saved new version from row in /hbase i can't
replace it from /hbase.old
Thanks for time
Lukas
2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> Lukas,
>
> A little trick you can use is to just copy the table directories into
> your new /hbase dir and then use:
>
> hbck -fixMeta -fixAssignments
>
> This will pull in the tables into your new empty META. If once you do this
> Regions get stuck in transition again, you have bigger problems.
>
> On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <lu...@gmail.com> wrote:
>
>> Hi again.
>>
>> > Does FSCK come back clean? Are those the regions showing in transition?
>> > We are not going to be able to get a clear idea of what to do next until
>> > we gather some more data. At this point running repairs could put your
>> > data in jeopardy.
>>
>>
>> Ye, this is in transition, but from hadoop fsck are all ok and
>> consistent - reported only 14 undereplicated blocks from another
>> directory (not /hbase).
>>
>> I must bring cluster back online, so i move /hbase on hdfs to
>> /hbase.old and start clear cluster with new tables :(
>>
>> Can you give me please any suggestion how i can put data from old
>> files to new tables?
>>
>> Thanks
>>
>> My today's night will be very long :(
>>
>> Lukas
>>
>> >
>> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <lu...@gmail.com>
>> wrote:
>> >
>> >> Kevin,
>> >>
>> >> Its more than one hour and region are still in transition.
>> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>> >>
>> >> Thanks for time
>> >>
>> >> Lukas
>> >>
>> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> >> > Lukas,
>> >> >
>> >> > Sure, how long has it been since the restart? You will need to give
>> >> the
>> >> > regions time to transition and logs time to split. Did the region
>> >> > in transition properly? Can you please out a pastebin together of
>> hbck
>> >> > -details full output so that I can take a look at it. Once I have
>> >> reviewed
>> >> > it, I should be able to provide you with a plan to out this back
>> >> together.
>> >> >
>> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Hello Kevin,
>> >> >>
>> >> >> thanks for response. I clear it now and start cluster, but now i have
>> >> >> many other's regions :(
>> >> >> Here is output from hbase hbck -fixmeta
>> >> >> http://pastebin.com/HqsPVLMi
>> >> >>
>> >> >> Any next hint?
>> >> >>
>> >> >> Thanks a lot
>> >> >>
>> >> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> >> >> > Have you tried clearing out your Znode information? Typically,
>> when I
>> >> >> have
>> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
>> >> clear
>> >> >> > out /hbase. What do you see in the logs pertaining to the region?
>> >> If it
>> >> >> > is a region that has bad hfiles or something like that you will
>> not be
>> >> >> able
>> >> >> > to easily get rid of it through clearing out the ZK.
>> >> >> >
>> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
>> lukas.drbal@gmail.com>
>> >> >> wrote:
>> >> >> >
>> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >> >> >>
>> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
>> >> line
>> >> >> >> are repeated for all region servers.
>> >> >> >>
>> >> >> >> Can someone help me please?
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Kevin O'Dell
>> >> >> > Customer Operations Engineer, Cloudera
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Save The World - http://www.worldcommunitygrid.org/
>> >> >>
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >> >>
>> >> >> LesTR
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kevin O'Dell
>> >> > Customer Operations Engineer, Cloudera
>> >>
>> >>
>> >>
>> >> --
>> >> Save The World - http://www.worldcommunitygrid.org/
>> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >>
>> >> LesTR
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>>
>>
>> --
>> Save The World - http://www.worldcommunitygrid.org/
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>>
>> LesTR
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
LesTR
Re: Region in Transition
Posted by Kevin O'dell <ke...@cloudera.com>.
Lukas,
A little trick you can use is to just copy the table directories into
your new /hbase dir and then use:
hbck -fixMeta -fixAssignments
This will pull in the tables into your new empty META. If once you do this
Regions get stuck in transition again, you have bigger problems.
On Mon, Oct 15, 2012 at 11:48 AM, Lukáš Drbal <lu...@gmail.com> wrote:
> Hi again.
>
> > Does FSCK come back clean? Are those the regions showing in transition?
> > We are not going to be able to get a clear idea of what to do next until
> > we gather some more data. At this point running repairs could put your
> > data in jeopardy.
>
>
> Ye, this is in transition, but from hadoop fsck are all ok and
> consistent - reported only 14 undereplicated blocks from another
> directory (not /hbase).
>
> I must bring cluster back online, so i move /hbase on hdfs to
> /hbase.old and start clear cluster with new tables :(
>
> Can you give me please any suggestion how i can put data from old
> files to new tables?
>
> Thanks
>
> My today's night will be very long :(
>
> Lukas
>
> >
> > On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <lu...@gmail.com>
> wrote:
> >
> >> Kevin,
> >>
> >> Its more than one hour and region are still in transition.
> >> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
> >>
> >> Thanks for time
> >>
> >> Lukas
> >>
> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> >> > Lukas,
> >> >
> >> > Sure, how long has it been since the restart? You will need to give
> >> the
> >> > regions time to transition and logs time to split. Did the region
> >> > in transition properly? Can you please out a pastebin together of
> hbck
> >> > -details full output so that I can take a look at it. Once I have
> >> reviewed
> >> > it, I should be able to provide you with a plan to out this back
> >> together.
> >> >
> >> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com>
> >> wrote:
> >> >
> >> >> Hello Kevin,
> >> >>
> >> >> thanks for response. I clear it now and start cluster, but now i have
> >> >> many other's regions :(
> >> >> Here is output from hbase hbck -fixmeta
> >> >> http://pastebin.com/HqsPVLMi
> >> >>
> >> >> Any next hint?
> >> >>
> >> >> Thanks a lot
> >> >>
> >> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> >> >> > Have you tried clearing out your Znode information? Typically,
> when I
> >> >> have
> >> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
> >> clear
> >> >> > out /hbase. What do you see in the logs pertaining to the region?
> >> If it
> >> >> > is a region that has bad hfiles or something like that you will
> not be
> >> >> able
> >> >> > to easily get rid of it through clearing out the ZK.
> >> >> >
> >> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <
> lukas.drbal@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >> >> >>
> >> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
> >> line
> >> >> >> are repeated for all region servers.
> >> >> >>
> >> >> >> Can someone help me please?
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Kevin O'Dell
> >> >> > Customer Operations Engineer, Cloudera
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Save The World - http://www.worldcommunitygrid.org/
> >> >>
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >> >>
> >> >> LesTR
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Kevin O'Dell
> >> > Customer Operations Engineer, Cloudera
> >>
> >>
> >>
> >> --
> >> Save The World - http://www.worldcommunitygrid.org/
> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >>
> >> LesTR
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
>
> --
> Save The World - http://www.worldcommunitygrid.org/
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>
> LesTR
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: Region in Transition
Posted by Lukáš Drbal <lu...@gmail.com>.
Hi again.
> Does FSCK come back clean? Are those the regions showing in transition?
> We are not going to be able to get a clear idea of what to do next until
> we gather some more data. At this point running repairs could put your
> data in jeopardy.
Ye, this is in transition, but from hadoop fsck are all ok and
consistent - reported only 14 undereplicated blocks from another
directory (not /hbase).
I must bring cluster back online, so i move /hbase on hdfs to
/hbase.old and start clear cluster with new tables :(
Can you give me please any suggestion how i can put data from old
files to new tables?
Thanks
My today's night will be very long :(
Lukas
>
> On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <lu...@gmail.com> wrote:
>
>> Kevin,
>>
>> Its more than one hour and region are still in transition.
>> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>>
>> Thanks for time
>>
>> Lukas
>>
>> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> > Lukas,
>> >
>> > Sure, how long has it been since the restart? You will need to give
>> the
>> > regions time to transition and logs time to split. Did the region
>> > in transition properly? Can you please out a pastebin together of hbck
>> > -details full output so that I can take a look at it. Once I have
>> reviewed
>> > it, I should be able to provide you with a plan to out this back
>> together.
>> >
>> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com>
>> wrote:
>> >
>> >> Hello Kevin,
>> >>
>> >> thanks for response. I clear it now and start cluster, but now i have
>> >> many other's regions :(
>> >> Here is output from hbase hbck -fixmeta
>> >> http://pastebin.com/HqsPVLMi
>> >>
>> >> Any next hint?
>> >>
>> >> Thanks a lot
>> >>
>> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> >> > Have you tried clearing out your Znode information? Typically, when I
>> >> have
>> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
>> clear
>> >> > out /hbase. What do you see in the logs pertaining to the region?
>> If it
>> >> > is a region that has bad hfiles or something like that you will not be
>> >> able
>> >> > to easily get rid of it through clearing out the ZK.
>> >> >
>> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >> >>
>> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
>> line
>> >> >> are repeated for all region servers.
>> >> >>
>> >> >> Can someone help me please?
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kevin O'Dell
>> >> > Customer Operations Engineer, Cloudera
>> >>
>> >>
>> >>
>> >> --
>> >> Save The World - http://www.worldcommunitygrid.org/
>> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>> >>
>> >> LesTR
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>>
>>
>> --
>> Save The World - http://www.worldcommunitygrid.org/
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>>
>> LesTR
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
LesTR
Re: Region in Transition
Posted by Kevin O'dell <ke...@cloudera.com>.
Lukas,
Looking over what you sent me:
ERROR: Region { meta =>
twitter_users,1ffd1a52913c3fd600dedb97d1b2b8ce,1350294703353.6e3be1c26233ed744479289b00a422dd.,
hdfs =>
hdfs://hadoop-7:9000/hbase/twitter_users/6e3be1c26233ed744479289b00a422dd,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,6103caf551a3115ee22e527542464a33,1350306570677.7112dd6fd61f2cf2675696c8320f7639.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/7112dd6fd61f2cf2675696c8320f7639,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,81a92d771ace1b49526b0c940fc65fb5,1350294695237.8f290a471d0cd0d1036b306fa526db4e.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/8f290a471d0cd0d1036b306fa526db4e,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,f01f68e30ffbeca7d248091f5319ffee,1350306575742.952fa87b758d1605bb263535376fdf33.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/952fa87b758d1605bb263535376fdf33,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
twitter_tweets,4fc3d2db81837d6aafb97348da86bcd7,1350308118451.a532656e5d112c2962648cd5598c635d.,
hdfs =>
hdfs://hadoop-7:9000/hbase/twitter_tweets/a532656e5d112c2962648cd5598c635d,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
twitter_users,afcb98b707ee4ee8a830827e8a7265ed,1350294705684.b222aa685d8a53ba836518074b7e27a0.,
hdfs =>
hdfs://hadoop-7:9000/hbase/twitter_users/b222aa685d8a53ba836518074b7e27a0,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
facebook_comments,e0405892a3d2065201252311ece8b5f9,1350306514155.c60bdbfcba96d4b381d7470756b15fdd.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_comments/c60bdbfcba96d4b381d7470756b15fdd,
deployed => } not deployed on any region server.
ERROR: Region { meta =>
facebook_posts,20600f7f12eb2d850c4e75d7e6dcbb84,1350294692505.d0192a38206a8df60210f6d972807f64.,
hdfs =>
hdfs://hadoop-7:9000/hbase/facebook_posts/d0192a38206a8df60210f6d972807f64,
deployed => } not deployed on any region server.
Does FSCK come back clean? Are those the regions showing in transition?
We are not going to be able to get a clear idea of what to do next until
we gather some more data. At this point running repairs could put your
data in jeopardy.
On Mon, Oct 15, 2012 at 10:58 AM, Lukáš Drbal <lu...@gmail.com> wrote:
> Kevin,
>
> Its more than one hour and region are still in transition.
> Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
>
> Thanks for time
>
> Lukas
>
> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> > Lukas,
> >
> > Sure, how long has it been since the restart? You will need to give
> the
> > regions time to transition and logs time to split. Did the region
> > in transition properly? Can you please out a pastebin together of hbck
> > -details full output so that I can take a look at it. Once I have
> reviewed
> > it, I should be able to provide you with a plan to out this back
> together.
> >
> > On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com>
> wrote:
> >
> >> Hello Kevin,
> >>
> >> thanks for response. I clear it now and start cluster, but now i have
> >> many other's regions :(
> >> Here is output from hbase hbck -fixmeta
> >> http://pastebin.com/HqsPVLMi
> >>
> >> Any next hint?
> >>
> >> Thanks a lot
> >>
> >> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> >> > Have you tried clearing out your Znode information? Typically, when I
> >> have
> >> > encountered a RIT, we will bring down HBase and go to the ZKcli and
> clear
> >> > out /hbase. What do you see in the logs pertaining to the region?
> If it
> >> > is a region that has bad hfiles or something like that you will not be
> >> able
> >> > to easily get rid of it through clearing out the ZK.
> >> >
> >> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com>
> >> wrote:
> >> >
> >> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >> >>
> >> >> I found this in master log http://pastebin.com/CYt5PZCL and this
> line
> >> >> are repeated for all region servers.
> >> >>
> >> >> Can someone help me please?
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Kevin O'Dell
> >> > Customer Operations Engineer, Cloudera
> >>
> >>
> >>
> >> --
> >> Save The World - http://www.worldcommunitygrid.org/
> >> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
> >>
> >> LesTR
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
>
> --
> Save The World - http://www.worldcommunitygrid.org/
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>
> LesTR
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: Region in Transition
Posted by Lukáš Drbal <lu...@gmail.com>.
Kevin,
Its more than one hour and region are still in transition.
Here is output from hbck -detailed http://pastebin.com/f9wLx9LU
Thanks for time
Lukas
2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> Lukas,
>
> Sure, how long has it been since the restart? You will need to give the
> regions time to transition and logs time to split. Did the region
> in transition properly? Can you please out a pastebin together of hbck
> -details full output so that I can take a look at it. Once I have reviewed
> it, I should be able to provide you with a plan to out this back together.
>
> On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com> wrote:
>
>> Hello Kevin,
>>
>> thanks for response. I clear it now and start cluster, but now i have
>> many other's regions :(
>> Here is output from hbase hbck -fixmeta
>> http://pastebin.com/HqsPVLMi
>>
>> Any next hint?
>>
>> Thanks a lot
>>
>> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
>> > Have you tried clearing out your Znode information? Typically, when I
>> have
>> > encountered a RIT, we will bring down HBase and go to the ZKcli and clear
>> > out /hbase. What do you see in the logs pertaining to the region? If it
>> > is a region that has bad hfiles or something like that you will not be
>> able
>> > to easily get rid of it through clearing out the ZK.
>> >
>> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com>
>> wrote:
>> >
>> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>> >>
>> >> I found this in master log http://pastebin.com/CYt5PZCL and this line
>> >> are repeated for all region servers.
>> >>
>> >> Can someone help me please?
>> >>
>> >
>> >
>> >
>> > --
>> > Kevin O'Dell
>> > Customer Operations Engineer, Cloudera
>>
>>
>>
>> --
>> Save The World - http://www.worldcommunitygrid.org/
>> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>>
>> LesTR
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
LesTR
Re: Region in Transition
Posted by Kevin O'dell <ke...@cloudera.com>.
Lukas,
Sure, how long has it been since the restart? You will need to give the
regions time to transition and logs time to split. Did the region
in transition properly? Can you please out a pastebin together of hbck
-details full output so that I can take a look at it. Once I have reviewed
it, I should be able to provide you with a plan to out this back together.
On Mon, Oct 15, 2012 at 10:24 AM, Lukáš Drbal <lu...@gmail.com> wrote:
> Hello Kevin,
>
> thanks for response. I clear it now and start cluster, but now i have
> many other's regions :(
> Here is output from hbase hbck -fixmeta
> http://pastebin.com/HqsPVLMi
>
> Any next hint?
>
> Thanks a lot
>
> 2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> > Have you tried clearing out your Znode information? Typically, when I
> have
> > encountered a RIT, we will bring down HBase and go to the ZKcli and clear
> > out /hbase. What do you see in the logs pertaining to the region? If it
> > is a region that has bad hfiles or something like that you will not be
> able
> > to easily get rid of it through clearing out the ZK.
> >
> > On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com>
> wrote:
> >
> >> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
> >>
> >> I found this in master log http://pastebin.com/CYt5PZCL and this line
> >> are repeated for all region servers.
> >>
> >> Can someone help me please?
> >>
> >
> >
> >
> > --
> > Kevin O'Dell
> > Customer Operations Engineer, Cloudera
>
>
>
> --
> Save The World - http://www.worldcommunitygrid.org/
> http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
>
> LesTR
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: Region in Transition
Posted by Lukáš Drbal <lu...@gmail.com>.
Hello Kevin,
thanks for response. I clear it now and start cluster, but now i have
many other's regions :(
Here is output from hbase hbck -fixmeta
http://pastebin.com/HqsPVLMi
Any next hint?
Thanks a lot
2012/10/15 Kevin O'dell <ke...@cloudera.com>:
> Have you tried clearing out your Znode information? Typically, when I have
> encountered a RIT, we will bring down HBase and go to the ZKcli and clear
> out /hbase. What do you see in the logs pertaining to the region? If it
> is a region that has bad hfiles or something like that you will not be able
> to easily get rid of it through clearing out the ZK.
>
> On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com> wrote:
>
>> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>>
>> I found this in master log http://pastebin.com/CYt5PZCL and this line
>> are repeated for all region servers.
>>
>> Can someone help me please?
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
--
Save The World - http://www.worldcommunitygrid.org/
http://www.worldcommunitygrid.org/stat/viewMemberInfo.do?userName=LesTR
LesTR
Re: Region in Transition
Posted by Kevin O'dell <ke...@cloudera.com>.
Have you tried clearing out your Znode information? Typically, when I have
encountered a RIT, we will bring down HBase and go to the ZKcli and clear
out /hbase. What do you see in the logs pertaining to the region? If it
is a region that has bad hfiles or something like that you will not be able
to easily get rid of it through clearing out the ZK.
On Mon, Oct 15, 2012 at 8:59 AM, Lukáš Drbal <lu...@gmail.com> wrote:
> Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
>
> I found this in master log http://pastebin.com/CYt5PZCL and this line
> are repeated for all region servers.
>
> Can someone help me please?
>
--
Kevin O'Dell
Customer Operations Engineer, Cloudera
Re: Region in Transition
Posted by Lukáš Drbal <lu...@gmail.com>.
Oh sorry. My HBase version is 0.94.1 and hadoop 1.0.3
I found this in master log http://pastebin.com/CYt5PZCL and this line
are repeated for all region servers.
Can someone help me please?