[jira] [Resolved] (HBASE-29774) incremental backup fails on empty WAL files

["Dieter De Paepe (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-29774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dieter De Paepe resolved HBASE-29774.
-------------------------------------
    Resolution: Fixed

> incremental backup fails on empty WAL files
> -------------------------------------------
>
>                 Key: HBASE-29774
>                 URL: https://issues.apache.org/jira/browse/HBASE-29774
>             Project: HBase
>          Issue Type: Bug
>          Components: backup&restore
>    Affects Versions: 2.6.4
>            Reporter: Jan Van Besien
>            Assignee: Dieter De Paepe
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0-alpha-1, 3.0.0-beta-2, 2.6.7
>
>
> Incremental backup fails during {{IncrementalTableBackupClient.convertWALsToHFiles}} when one of these WAL files is empty (zero bytes). 
> The map tasks fail like this:
> {code:java}
> 2025-12-12 06:22:21,785 INFO [main] org.apache.hadoop.hbase.mapreduce.WALInputFormat: Opening hdfs://hdfsns/hbase/hbase/oldWALs/hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.regiongroup-0.1764754214409 for hdfs://hdfsns/hbase/hbase/oldWALs/hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.hbase-region-0.hbase-region.qa03-shared.svc.cluster.local%2C16020%2C1764740248035.regiongroup-0.1764754214409 (-9223372036854775808:9223372036854775807) length:0
> 2025-12-12 06:22:21,810 INFO [main] org.apache.hadoop.hbase.mapreduce.WALInputFormat: Closing reader
> 2025-12-12 06:22:21,811 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
> 2025-12-12 06:22:21,815 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
> 2025-12-12 06:22:21,904 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.hbase.regionserver.wal.WALHeaderEOFException: EOF while reading PB WAL magic
> 	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader.readHeader(AbstractProtobufWALReader.java:221)
> 	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader.init(AbstractProtobufWALReader.java:147)
> 	at org.apache.hadoop.hbase.wal.WALFactory.createStreamReader(WALFactory.java:360)
> 	at org.apache.hadoop.hbase.wal.WALFactory.createStreamReader(WALFactory.java:481)
> 	at org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.openReader(WALInputFormat.java:162)
> 	at org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.openReader(WALInputFormat.java:204)
> 	at org.apache.hadoop.hbase.mapreduce.WALInputFormat$WALRecordReader.initialize(WALInputFormat.java:197)
> 	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:561)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
> 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
> 	at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
> Caused by: java.io.EOFException
> 	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:210)
> 	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:179)
> 	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufWALReader.readHeader(AbstractProtobufWALReader.java:219)
> 	... 14 more
> {code}
> The file mentioned in the above log snippet is indeed zero bytes.
> The calling code fails like this:
> {code}
> 2025-12-12 06:22:45,140 ERROR org.apache.hadoop.hbase.backup.impl.TableBackupClient: Unexpected exception in incremental-backup: incremental copy backup_1765519365442WAL Player failed
> java.io.IOException: WAL Player failed
>         at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.walToHFiles(IncrementalTableBackupClient.java:448)
>         at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.convertWALsToHFiles(IncrementalTableBackupClient.java:414)
>         at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:311)
>         at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:594)
> {code}
> The javadoc in {{WALHeaderEOFException}} says "This usually means the WAL file just contains nothing and we are safe to skip over it.". So maybe that is indeed what needs to happen here?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)