[ https://issues.apache.org/jira/browse/HBASE-29111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Duo Zhang resolved HBASE-29111.
-------------------------------
Fix Version/s: 2.7.0
3.0.0-beta-2
2.6.2
2.5.12
Hadoop Flags: Reviewed
Resolution: Fixed
Pushed to all active branches.
Thanks [~junegunn] for contributing!
> Data loss in table cloned from a snapshot
> -----------------------------------------
>
> Key: HBASE-29111
> URL: https://issues.apache.org/jira/browse/HBASE-29111
> Project: HBase
> Issue Type: Bug
> Components: dataloss, snapshots
> Reporter: Junegunn Choi
> Assignee: Junegunn Choi
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.2, 2.5.12
>
>
> We experienced permanent data loss in a table cloned from a snapshot.
> Here's what we found.
> * If you clone a table from a snapshot that contains split regions and reference files, and immediately delete the snapshot, HBase can prematurely delete the original HFiles causing data loss.
> h2. How to reproduce
> To quickly reproduce the issue, adjust the cleaner and janitor intervals and set HFile TTL to zero. Also disable compaction so that reference files are not compacted away. Or, you can put a lot of data so that compaction doesn't finish during the test.
> {code:java}
> <property>
> <name>hbase.master.cleaner.interval</name>
> <value>1000</value>
> </property>
> <property>
> <name>hbase.catalogjanitor.interval</name>
> <value>1000</value>
> </property>
> <property>
> <name>hbase.master.hfilecleaner.ttl</name>
> <value>0</value>
> </property>
> <property>
> <name>hbase.regionserver.compaction.enabled</name>
> <value>false</value>
> </property>
> {code}
> And run this code on HBase shell.
> {code:java}
> # Create test table and write some data
> create 't', 'd'
> 10.times do |i|
> put 't', i, 'd:foo', '_' * 1024
> end
> # Split in the middle and take the snapshot
> split 't', '5'
> snapshot 't', 's'
> # Drop the table and clone it from the snapshot
> disable 't'
> drop 't'
> clone_snapshot 's', 't'
> # Immediately delete the snapshot
> delete_snapshot 's'
> # Try disabling and re-enabling the table
> sleep 2
> disable 't'
> enable 't'
> # java.io.FileNotFoundException: HFileLink locations=[...]
> {code}
> h2. What actually happens
> User clones a table from a snapshot containing split regions and reference files.
> {noformat}
> snapshot.RestoreSnapshotHelper: clone region=a23be88470c13611f6f24f20e0cf00ed as a23be88470c13611f6f24f20e0cf00ed in snapshot s
> ...
> regionserver.HRegion: creating {ENCODED => a23be88470c13611f6f24f20e0cf00ed, NAME => 't,40000000,1738562472443.a23be88470c13611f6f24f20e0cf00ed.', STARTKEY => '40000000', ENDKEY => '80000000', OFFLINE => true, SPLIT => true}, tableDescriptor='t', {TABLE_ATTRIBUTES => {METADATA => {'hbase.store.file-tracker.impl' => 'DEFAULT'}}}, {NAME => 'd', INDEX_BLOCK_ENCODING => 'NONE', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536 B (64KB)'}, regionDir=file:/Users/jg/github/hbase/tmp/hbase
> ...
> snapshot.RestoreSnapshotHelper: finishing restore table regions using snapshot=name: "s"
> {noformat}
> And the user deletes the snapshot.
> {noformat}
> snapshot.SnapshotManager: Deleting snapshot: s
> {noformat}
> After a while, CatalogJanitor garbage-collects the split parents as it sees no daughter information in the meta table.
> {noformat}
> janitor.CatalogJanitor: Cleaning parent region {ENCODED => a23be88470c13611f6f24f20e0cf00ed, NAME => 't,40000000,1738562472443.a23be88470c13611f6f24f20e0cf00ed.', STARTKEY => '40000000', ENDKEY => '80000000', OFFLINE => true, SPLIT => true}
> janitor.CatalogJanitor: Deleting region a23be88470c13611f6f24f20e0cf00ed because daughters -- null, null -- no longer hold references
> {noformat}
> (see "{{{}null, null"{}}} part)
> This causes the HFileLinks to be archived.
> {noformat}
> backup.HFileArchiver: Archived from FileablePath, file:/Users/jg/github/hbase/tmp/hbase/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/t=a23be88470c13611f6f24f20e0cf00ed-ecdd6aa22a6146599467839c56767522 to file:/Users/jg/github/hbase/tmp/hbase/archive/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/t=a23be88470c13611f6f24f20e0cf00ed-ecdd6aa22a6146599467839c56767522
> {noformat}
> And the cleaners unanimously agree to delete the original HFile.
> * We already deleted the snapshot, so SnapshotHFileCleaner won't complain.
> * Because HFileLink is archived, HFileLinkCleaner won't complain.
> And the HFile is deleted before the daughter regions succeed to rebuild the data in it through compaction.
> {noformat}
> cleaner.HFileCleaner: Removing file:/Users/jg/github/hbase/tmp/hbase/archive/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/ecdd6aa22a6146599467839c56767522
> {noformat}
> And the data loss.
> {noformat}
> regionserver.CompactSplit: Compaction selection failed region=t,6,1738562566034.8ad3785a3afe89e59b72db5d5d3a1bf5., storeName=8ad3785a3afe89e59b72db5d5d3a1bf5/d, priority=14, startTime=1738562622689
> java.io.FileNotFoundException: HFileLink locations=[
> file:/Users/jg/github/hbase/tmp/hbase/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/ecdd6aa22a6146599467839c56767522,
> file:/Users/jg/github/hbase/tmp/hbase/archive/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/ecdd6aa22a6146599467839c56767522,
> file:/Users/jg/github/hbase/tmp/hbase/.tmp/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/ecdd6aa22a6146599467839c56767522,
> file:/Users/jg/github/hbase/tmp/hbase/mobdir/data/default/t/a23be88470c13611f6f24f20e0cf00ed/d/ecdd6aa22a6146599467839c56767522
> ]
> {noformat}
> h2. Fix
> Make sure to put the split information to the meta table when cloning a table. The information was already there, we just didn't use it.
> Let me open pull requests both on master and branch-2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)