You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Bram Schuur (Jira)" <ji...@apache.org> on 2025/12/17 15:27:00 UTC
[jira] [Created] (HBASE-29784) DeleteFamilyVersion is not effectuated even though it is committed to WAL
Bram Schuur created HBASE-29784:
-----------------------------------
Summary: DeleteFamilyVersion is not effectuated even though it is committed to WAL
Key: HBASE-29784
URL: https://issues.apache.org/jira/browse/HBASE-29784
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.6.3
Environment: JDK: 21.0.9
HBAse: 2.6.3
Hadoop: 3.4.2
Arch: x86
OS: Containerized linux
Reporter: Bram Schuur
We are running hbase 2.6.3 as a datastore, sometimes we wipe data through DeleteFamilyVersion. Every now and then (intermittent, non-deterministic), the hbase database somehow forgets about a 'DeleteFamilyVersion' that we emitted for a row, making the data we meant to erase to appear again.
We started capturing more extensive WAL logs for our regions, which shows the DeleteFamilyVersion we emit is committed to WAL, however the data is still visible through the api after flushing/compaction of the region. There are no errors in the logs.
Below a snippet of the data we traced:
Data as queried from the hbase api:
{code}
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:\x00/1765693071241000000/Put/vlen=1/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:description/1765693071241000000/Put/vlen=4/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:domainIdentifier/1765693071241000000/Put/vlen=60/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:domainName/1765693071241000000/Put/vlen=8/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:identifiers/1765693071241000000/Put/vlen=94/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:lastUpdateTimestamp/1765693071241000000/Put/vlen=8/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:layerIdentifier/1765693071241000000/Put/vlen=39/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:layerName/1765693071241000000/Put/vlen=12/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:name/1765693071241000000/Put/vlen=26/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:order/1765693071241000000/Put/vlen=10/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:properties/1765693071241000000/Put/vlen=208/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:tags/1765693071241000000/Put/vlen=184/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:typeIdentifier/1765693071241000000/Put/vlen=72/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:typeName/1765693071241000000/Put/vlen=10/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:~\x00/1765693071241000000/Put/vlen=11/seqid=0
\x00\x00\xC7\xEBs\xFB\xCA\xDA/cf:~e\x06SYNCED\x00\x00\xDA\xB7\xE2\xD3\x8FW/1765693071241000000/Put/vlen=16/seqid=0
{code}
Data in captured WAL:
{code}
...
Sequence=10628094, table=sg__default__vertices, region=834ed0ff02e8d7d42b88ad5666a4b1e8, at write timestamp=Sun Dec 14 06:17:51 UTC 2025
...
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:~\x00, timestamp=1765693071241000000, type=Put
value: \x03\x01Componen\xF4
cell total size sum: 96
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:domainIdentifier, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01urn:stackpack:stackstate-k8s-agent-v2:shared:domain:agen\xF4
cell total size sum: 160
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:identifiers, timestamp=1765693071241000000, type=Put
value: \x02!\x01\x01\x03\x01\xD7\x01urn:process:/i-06fb48dc80ed9944b-preprod-dev.preprod.stackstate.io:68116:1765692998000
cell total size sum: 184
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:domainName, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01Agen\xF4
cell total size sum: 96
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:typeName, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01proces\xF3
cell total size sum: 96
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:name, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01containerd-shim-runc-v\xB2
cell total size sum: 112
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:description, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01\x81
cell total size sum: 96
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:typeIdentifier, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01\xC4\x01urn:stackpack:stackstate-k8s-agent-v2:shared:component-type:process
cell total size sum: 168
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:layerIdentifier, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01urn:stackpack:common:layer:processe\xF3
cell total size sum: 136
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:layerName, timestamp=1765693071241000000, type=Put
value: \x02\x03\x01Processe\xF3
cell total size sum: 104
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:properties, timestamp=1765693071241000000, type=Put
value: \x02 \x01\x04\x03\x01hos\xF4\x03\x01i-06fb48dc80ed9944b-preprod-dev.preprod.stackstate.i\xEF\x03\x01external_i\xE4\x03\x01\xD7\x01urn:process:/i-06fb48dc80ed9944b-preprod-dev.preprod.stackstate.io:68116:1765692998000\x03\x01pi\xE4\x03\x016811\xB6\x03\x01create_tim\xE5\x03\x01176569299800\xB0
cell total size sum: 296
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:tags, timestamp=1765693071241000000, type=Put
value: \x02!\x01\x07\x03\x01host:i-06fb48dc80ed9944b-preprod-dev.preprod.stackstate.i\xEF\x03\x01stackpack:agen\xF4\x03\x01pid:6811\xB6\x03\x01user:roo\xF4\x03\x01os:linu\xF8\x03\x01command:/usr/bin/containerd-shim-runc-v\xB2\x03\x01process_category:executabl\xE5
cell total size sum: 272
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:order, timestamp=1765693071241000000, type=Put
value: \x02\x0A\x00\x00\x00\x00\x00\x00\x00\x00
cell total size sum: 96
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:lastUpdateTimestamp, timestamp=1765693071241000000, type=Put
value: \x02\x09\x92\xFE\x90\xB8\xE3f
cell total size sum: 112
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:~e\x06SYNCED\x00\x00\xDA\xB7\xE2\xD3\x8FW, timestamp=1765693071241000000, type=Put
value: \x01\x06Synced\x00\x00x\x87\xBA\xDE\xF7\xFE
cell total size sum: 112
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:\x00, timestamp=1765693071241000000, type=Put
value: \x01
cell total size sum: 80
...
position: 1481623
...
Sequence=10628100, table=sg__default__vertices, region=834ed0ff02e8d7d42b88ad5666a4b1e8, at write timestamp=Sun Dec 14 06:17:51 UTC 2025
...
row=\x00\x00\xC7\xEBs\xFB\xCA\xDA, column=cf:, timestamp=1765693071241000000, type=DeleteFamilyVersion
value:
cell total size sum: 80
...
position: 1531651
{code}
What could be the cause? I check the bugtracker but found nothing resembling/matching our symptomps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)