You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Omri Cohen <Om...@outlook.com> on 2019/08/17 13:28:59 UTC
Mob reference tags go missing
Hello,
We recently encountered a problem in our production hbase cluster.
We have a cluster with a high mob percentage ( > 50% of objects are MOB, the threshold is the default 102400 bytes). The cluster has been active for about a year. Recently, our clients started to receive the mob reference instead of the mob data when trying to access certain rows (in most tables about 10% of the rows are bad, in one table it is about 50% of rows).
When we investigated the HFiles, we saw that the "bad" cells are missing two tags that exist in the "good" rows, it looks something like that:
Re: Mob reference tags go missing
Posted by Sean Busbey <se...@gmail.com>.
Also the hfile version setting and hbase version on your masters, presuming
you are using the mob compaction chore in the master.
On Sat, Aug 17, 2019, 09:59 Sean Busbey <se...@gmail.com> wrote:
> That sounds like some of your hfiles are being written as hfile v2 instead
> of v3. Can you check versions and configs on all your region servers?
>
> On Sat, Aug 17, 2019, 08:42 Omri Cohen <Om...@outlook.com> wrote:
>
>> Sorry, accidentally pressed "send" before i finished the massage.
>>
>> Hello,
>>
>> We recently encountered a problem in our production hbase cluster (CDH
>> deployment, version 5.13.1).
>>
>> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
>> the threshold is the default 102400 bytes). The cluster has been active for
>> about a year. Recently, our clients started to receive the mob reference
>> instead of the mob data when trying to access certain rows (in most tables
>> about 10% of the rows are bad, in one table it is about 50% of rows).
>>
>> When we investigated the HFiles, we saw that the "bad" cells are missing
>> two tags that exist in the "good" rows, it looks something like that:
>>
>> K: {rowkey of a "good" row} {Column family, column qualifier}...
>> vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]: T[1]: {table name}
>> K: {rowkey of a "bad" row} {Column family, column qualifier}...
>> vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}
>>
>> The "good" row returns the expected data when queried, while the "bad"
>> row returned the mob file reference instead of the data. This happened when
>> we queried from the REST server, the native java client, and the hbase
>> shell.
>>
>> When we examined the mob files, we saw that all the data was there.
>>
>> * Has anyone encountered a similar situation?
>> * Is it possible to manually add the missing mob reference tags to
>> the "bad" rows?
>>
>> No changes where made to the table in recently. We never encountered this
>> issue before.
>>
>> We would appreciate any help on this issue.
>> ________________________________
>> From: Omri Cohen
>> Sent: Saturday, August 17, 2019 4:28 PM
>> To: hbase-user@hadoop.apache.org <hb...@hadoop.apache.org>
>> Subject: Mob reference tags go missing
>>
>> Hello,
>>
>> We recently encountered a problem in our production hbase cluster.
>> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
>> the threshold is the default 102400 bytes). The cluster has been active for
>> about a year. Recently, our clients started to receive the mob reference
>> instead of the mob data when trying to access certain rows (in most tables
>> about 10% of the rows are bad, in one table it is about 50% of rows).
>>
>> When we investigated the HFiles, we saw that the "bad" cells are missing
>> two tags that exist in the "good" rows, it looks something like that:
>>
>
Re: Mob reference tags go missing
Posted by Sean Busbey <se...@gmail.com>.
That sounds like some of your hfiles are being written as hfile v2 instead
of v3. Can you check versions and configs on all your region servers?
On Sat, Aug 17, 2019, 08:42 Omri Cohen <Om...@outlook.com> wrote:
> Sorry, accidentally pressed "send" before i finished the massage.
>
> Hello,
>
> We recently encountered a problem in our production hbase cluster (CDH
> deployment, version 5.13.1).
>
> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
> the threshold is the default 102400 bytes). The cluster has been active for
> about a year. Recently, our clients started to receive the mob reference
> instead of the mob data when trying to access certain rows (in most tables
> about 10% of the rows are bad, in one table it is about 50% of rows).
>
> When we investigated the HFiles, we saw that the "bad" cells are missing
> two tags that exist in the "good" rows, it looks something like that:
>
> K: {rowkey of a "good" row} {Column family, column qualifier}...
> vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]: T[1]: {table name}
> K: {rowkey of a "bad" row} {Column family, column qualifier}...
> vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}
>
> The "good" row returns the expected data when queried, while the "bad" row
> returned the mob file reference instead of the data. This happened when we
> queried from the REST server, the native java client, and the hbase shell.
>
> When we examined the mob files, we saw that all the data was there.
>
> * Has anyone encountered a similar situation?
> * Is it possible to manually add the missing mob reference tags to the
> "bad" rows?
>
> No changes where made to the table in recently. We never encountered this
> issue before.
>
> We would appreciate any help on this issue.
> ________________________________
> From: Omri Cohen
> Sent: Saturday, August 17, 2019 4:28 PM
> To: hbase-user@hadoop.apache.org <hb...@hadoop.apache.org>
> Subject: Mob reference tags go missing
>
> Hello,
>
> We recently encountered a problem in our production hbase cluster.
> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
> the threshold is the default 102400 bytes). The cluster has been active for
> about a year. Recently, our clients started to receive the mob reference
> instead of the mob data when trying to access certain rows (in most tables
> about 10% of the rows are bad, in one table it is about 50% of rows).
>
> When we investigated the HFiles, we saw that the "bad" cells are missing
> two tags that exist in the "good" rows, it looks something like that:
>
Re: Mob reference tags go missing
Posted by Omri Cohen <Om...@outlook.com>.
Sorry, accidentally pressed "send" before i finished the massage.
Hello,
We recently encountered a problem in our production hbase cluster (CDH deployment, version 5.13.1).
We have a cluster with a high mob percentage ( > 50% of objects are MOB, the threshold is the default 102400 bytes). The cluster has been active for about a year. Recently, our clients started to receive the mob reference instead of the mob data when trying to access certain rows (in most tables about 10% of the rows are bad, in one table it is about 50% of rows).
When we investigated the HFiles, we saw that the "bad" cells are missing two tags that exist in the "good" rows, it looks something like that:
K: {rowkey of a "good" row} {Column family, column qualifier}... vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]: T[1]: {table name}
K: {rowkey of a "bad" row} {Column family, column qualifier}... vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}
The "good" row returns the expected data when queried, while the "bad" row returned the mob file reference instead of the data. This happened when we queried from the REST server, the native java client, and the hbase shell.
When we examined the mob files, we saw that all the data was there.
* Has anyone encountered a similar situation?
* Is it possible to manually add the missing mob reference tags to the "bad" rows?
No changes where made to the table in recently. We never encountered this issue before.
We would appreciate any help on this issue.
________________________________
From: Omri Cohen
Sent: Saturday, August 17, 2019 4:28 PM
To: hbase-user@hadoop.apache.org <hb...@hadoop.apache.org>
Subject: Mob reference tags go missing
Hello,
We recently encountered a problem in our production hbase cluster.
We have a cluster with a high mob percentage ( > 50% of objects are MOB, the threshold is the default 102400 bytes). The cluster has been active for about a year. Recently, our clients started to receive the mob reference instead of the mob data when trying to access certain rows (in most tables about 10% of the rows are bad, in one table it is about 50% of rows).
When we investigated the HFiles, we saw that the "bad" cells are missing two tags that exist in the "good" rows, it looks something like that: