Empty REGIONINFO after Region Server Restart

[Mike Linsinbigler <ml...@ccri.com>]

Hello,

I am running HBase 1.4.10 on AWS EMR 5.29.0 and have had issues after 
some of our Region Servers restart (due to a variety of reasons) and the 
cluster enters an inconsistent state after they come back up.

Running hbck, I am presented with many instances of:

ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta


I am able to resolve this issue by running:

"hbase hbck -fixEmptyMetaCells"


However, this is only a fix until the next time one of our region 
servers restart which is currently a daily event. Does anyone know how 
to prevent this issue from occurring in the first place? It looks like 
our Region Server was in the middle of splitting and compaction 
operations before aborting.

I've noticed that writing to hbase while these errors are present can 
result in ingest issues within our application so I'd really like to 
understand how the meta table can get into this state.

Thanks,

Mike

Re: Empty REGIONINFO after Region Server Restart

[Zach York <zy...@gmail.com>]

This is definitely not expected behavior (either on S3 or HDFS).

I agree with Stack's request to try to trace how these are happening.
Since this is with EMR, you can also create a customer support ticket to
help debug the issue.

On Wed, Apr 22, 2020, 1:58 AM Wellington Chevreuil <
wellington.chevreuil@gmail.com> wrote:

> >
> > I am running HBase 1.4.10 on AWS EMR 5.29.0
> >
> Which file system is hbase root dir under? If it's S3, it's very likely
> some of its lack of consistency issues leading to missing regioninfo files.
>
> Em qua., 22 de abr. de 2020 às 02:21, Stack <st...@duboce.net> escreveu:
>
> > On Mon, Apr 20, 2020 at 7:59 AM Mike Linsinbigler <
> mlinsinbigler@ccri.com>
> > wrote:
> >
> > > Hello,
> > >
> > > I am running HBase 1.4.10 on AWS EMR 5.29.0 and have had issues after
> > > some of our Region Servers restart (due to a variety of reasons) and
> the
> > > cluster enters an inconsistent state after they come back up.
> > >
> > > Running hbck, I am presented with many instances of:
> > >
> > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
> > >
> > >
> > > I am able to resolve this issue by running:
> > >
> > > "hbase hbck -fixEmptyMetaCells"
> > >
> > >
> > > However, this is only a fix until the next time one of our region
> > > servers restart which is currently a daily event. Does anyone know how
> > > to prevent this issue from occurring in the first place? It looks like
> > > our Region Server was in the middle of splitting and compaction
> > > operations before aborting.
> > >
> > > I've noticed that writing to hbase while these errors are present can
> > > result in ingest issues within our application so I'd really like to
> > > understand how the meta table can get into this state.
> > >
> > >
> > Rows in hbase:meta w/ empty REGIONINFO_QUALIFIER will mess you up.
> Shoudn't
> > be happening. Have you tried tracing the lifecycle of one of these empty
> > rows? The row name is the region name. Search master logs over time using
> > the regionname. See if you can get a sense of what is happening
> > manufacturing empty rows.
> >
> > Yours,
> > S
> >
> >
> >
> > > Thanks,
> > >
> > > Mike
> > >
> > >
> >
>

Re: Empty REGIONINFO after Region Server Restart

[Wellington Chevreuil <we...@gmail.com>]

>
> I am running HBase 1.4.10 on AWS EMR 5.29.0
>
Which file system is hbase root dir under? If it's S3, it's very likely
some of its lack of consistency issues leading to missing regioninfo files.

Em qua., 22 de abr. de 2020 às 02:21, Stack <st...@duboce.net> escreveu:

> On Mon, Apr 20, 2020 at 7:59 AM Mike Linsinbigler <ml...@ccri.com>
> wrote:
>
> > Hello,
> >
> > I am running HBase 1.4.10 on AWS EMR 5.29.0 and have had issues after
> > some of our Region Servers restart (due to a variety of reasons) and the
> > cluster enters an inconsistent state after they come back up.
> >
> > Running hbck, I am presented with many instances of:
> >
> > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
> >
> >
> > I am able to resolve this issue by running:
> >
> > "hbase hbck -fixEmptyMetaCells"
> >
> >
> > However, this is only a fix until the next time one of our region
> > servers restart which is currently a daily event. Does anyone know how
> > to prevent this issue from occurring in the first place? It looks like
> > our Region Server was in the middle of splitting and compaction
> > operations before aborting.
> >
> > I've noticed that writing to hbase while these errors are present can
> > result in ingest issues within our application so I'd really like to
> > understand how the meta table can get into this state.
> >
> >
> Rows in hbase:meta w/ empty REGIONINFO_QUALIFIER will mess you up. Shoudn't
> be happening. Have you tried tracing the lifecycle of one of these empty
> rows? The row name is the region name. Search master logs over time using
> the regionname. See if you can get a sense of what is happening
> manufacturing empty rows.
>
> Yours,
> S
>
>
>
> > Thanks,
> >
> > Mike
> >
> >
>

Re: Empty REGIONINFO after Region Server Restart

[Stack <st...@duboce.net>]

On Mon, Apr 20, 2020 at 7:59 AM Mike Linsinbigler <ml...@ccri.com>
wrote:

> Hello,
>
> I am running HBase 1.4.10 on AWS EMR 5.29.0 and have had issues after
> some of our Region Servers restart (due to a variety of reasons) and the
> cluster enters an inconsistent state after they come back up.
>
> Running hbck, I am presented with many instances of:
>
> ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
>
>
> I am able to resolve this issue by running:
>
> "hbase hbck -fixEmptyMetaCells"
>
>
> However, this is only a fix until the next time one of our region
> servers restart which is currently a daily event. Does anyone know how
> to prevent this issue from occurring in the first place? It looks like
> our Region Server was in the middle of splitting and compaction
> operations before aborting.
>
> I've noticed that writing to hbase while these errors are present can
> result in ingest issues within our application so I'd really like to
> understand how the meta table can get into this state.
>
>
Rows in hbase:meta w/ empty REGIONINFO_QUALIFIER will mess you up. Shoudn't
be happening. Have you tried tracing the lifecycle of one of these empty
rows? The row name is the region name. Search master logs over time using
the regionname. See if you can get a sense of what is happening
manufacturing empty rows.

Yours,
S



> Thanks,
>
> Mike
>
>