You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Yashaswini G A (Jira)" <ji...@apache.org> on 2026/07/15 05:19:00 UTC
[jira] [Updated] (HDDS-15862) Snapshot incremental defrag fails at scale with - External file version not found (~5k snapshots stuck with needsDefrag=true)
[ https://issues.apache.org/jira/browse/HDDS-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yashaswini G A updated HDDS-15862:
----------------------------------
Description:
During snapshot defrag boundary QE testing (~10k snapshots on a single bucket), OM snapshot defrag fails to drain the needsDefrag backlog. Approximately half of snapshot checkpoint yaml entries remain stuck at needsDefrag=true.
OM logs error stack
{code:java}
2026-07-13 03:43:16,516 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Defragmenting snapshot: /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441 (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:16,516 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
2026-07-13 03:43:16,517 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,659 INFO SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager: Created checkpoint in rocksDB at /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590 in 68 milliseconds
2026-07-13 03:43:16,659 INFO SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils: Waited for 0 milliseconds for checkpoint directory /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590 availability.
2026-07-13 03:43:16,660 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,893 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,117 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Performing incremental defragmentation for snapshot: /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441 (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:17,348 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
2026-07-13 03:43:17,349 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,420 ERROR SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption: RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]: Failed to Failed to ingest external files /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst of fileTable
at org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
at org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
at org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
at org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.rocksdb.RocksDBException: An external sst file with version 2 have global seqno property with value ^A, while largest seqno in the file is 0
at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
... 14 more{code}
was:
During snapshot defrag boundary QE testing (~10k snapshots on a single bucket), OM snapshot defrag fails to drain the needsDefrag backlog. Approximately half of snapshot checkpoint yaml entries remain stuck at needsDefrag=true.
OM logs error stack
2026-07-13 03:43:16,516 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Defragmenting snapshot: /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441 (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:16,516 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
2026-07-13 03:43:16,517 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,659 INFO [SnapshotDefragService#0]-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager: Created checkpoint in rocksDB at /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590 in 68 milliseconds
2026-07-13 03:43:16,659 INFO [SnapshotDefragService#0]-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils: Waited for 0 milliseconds for checkpoint directory /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590 availability.
2026-07-13 03:43:16,660 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,893 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,117 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Performing incremental defragmentation for snapshot: /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441 (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:17,348 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
2026-07-13 03:43:17,349 INFO [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,420 ERROR [SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption: RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]: Failed to Failed to ingest external files /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst of fileTable
at org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
at org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
at org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
at org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.rocksdb.RocksDBException: An external sst file with version 2 have global seqno property with value ^A, while largest seqno in the file is 0
at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
at org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
... 14 more
> Snapshot incremental defrag fails at scale with - External file version not found (~5k snapshots stuck with needsDefrag=true)
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15862
> URL: https://issues.apache.org/jira/browse/HDDS-15862
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Yashaswini G A
> Priority: Major
>
> During snapshot defrag boundary QE testing (~10k snapshots on a single bucket), OM snapshot defrag fails to drain the needsDefrag backlog. Approximately half of snapshot checkpoint yaml entries remain stuck at needsDefrag=true.
> OM logs error stack
> {code:java}
> 2026-07-13 03:43:16,516 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Defragmenting snapshot: /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441 (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
> 2026-07-13 03:43:16,516 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
> 2026-07-13 03:43:16,517 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:16,659 INFO SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager: Created checkpoint in rocksDB at /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590 in 68 milliseconds
> 2026-07-13 03:43:16,659 INFO SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils: Waited for 0 milliseconds for checkpoint directory /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590 availability.
> 2026-07-13 03:43:16,660 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:16,893 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:17,117 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Performing incremental defragmentation for snapshot: /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441 (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
> 2026-07-13 03:43:17,348 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
> 2026-07-13 03:43:17,349 INFO SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:17,420 ERROR SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService: Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
> org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption: RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]: Failed to Failed to ingest external files /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst of fileTable
> at org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
> at org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
> at org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
> at org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
> at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
> at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
> at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
> at org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
> at org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
> at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: org.rocksdb.RocksDBException: An external sst file with version 2 have global seqno property with value ^A, while largest seqno in the file is 0
> at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
> at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
> at org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
> ... 14 more{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org