[jira] [Commented] (HBASE-28287) MOB HFiles are expired earlier than their reference data

["WenjingLiu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-28287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800318#comment-17800318 ] 

WenjingLiu commented on HBASE-28287:
------------------------------------

Hi [~zhangduo] ,
Here is the exception when I read mob data from a Time-To-Live table:

hbase(main):003:0> get 'tbl1_1days', 'rowkey1'
COLUMN                                                      CELL                                                                                                                                                                          

ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: \{HBASE_DIR}/archive/\{namespace}/\{tableName}/\{regionEncodedName}/\{columnFamilyName}/\{mobFileName}
        at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:434)
        at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:348)
        at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:73)
        at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:178)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6636)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6824)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6573)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6550)
        at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:6537)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2599)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2520)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Caused by: java.io.FileNotFoundException: File does not exist: \{HBASE_DIR}/archive/\{namespace}/\{tableName}/\{regionEncodedName}/\{columnFamilyName}/\{mobFileName}
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1587)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
        at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:139)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:220)
        at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
        at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:221)
        at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:395)
        ... 15 more

For usage try 'help "get"'

Took 1.2261 seconds 

Looking forward to your feedback.

> MOB HFiles are expired earlier than their reference data
> --------------------------------------------------------
>
>                 Key: HBASE-28287
>                 URL: https://issues.apache.org/jira/browse/HBASE-28287
>             Project: HBase
>          Issue Type: Bug
>          Components: mob
>    Affects Versions: 2.5.0
>            Reporter: WenjingLiu
>            Priority: Major
>
> I have observed that mob HFiles are expired earlier than their reference data. Upon reviewing the relevant code, it was observed that the standard expired timestamp is not accurately set. The issue arises from the fact that if we do not set the calendar to the MILLISECOND level, a three-digit random number is appended to the end of the expired timestamp, leading to inaccuracies.
> For instance, if we write mob data to a Time-To-Live (1 day) table at 22:00 on 12/20/2023, and the ExpiredMobFileCleaner begins to work at 20:00 on 12/21/2023, the timestamp of the mob HFile will be parsed as 1703001600000, and the StandardExpiredTS is calculated as 1703001600314 with the three-digit random number as MILLISECOND. Consequently, the StandardExpiredTS >= MobHfileTS, thus indicating that the mob HFile should be expired. However, it is observed that this mob HFile only exists for 22 hours, which is less than the specified one-day-time threshold.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)