You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Hemant Kumar (Jira)" <ji...@apache.org> on 2023/08/15 23:14:00 UTC
[jira] [Commented] (HDDS-8940) SST files are missing on optimized snapDiff path.
[ https://issues.apache.org/jira/browse/HDDS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754810#comment-17754810 ]
Hemant Kumar commented on HDDS-8940:
------------------------------------
There is another occurrence of this issue.
Important logs.
{code}
...
2023-08-14 19:29:45,520 INFO [OM StateMachine ApplyTransaction Thread - 0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created snapshot: 'cm-46-1692041157584-0' with snapshotId: 'c2ae0246-626b-4a09-b1c8-0e76997c26ee' under path 'vol23io/bucket870io'
...
...
2023-08-14 19:30:25,363 INFO [SstFilteringService#0]-org.apache.hadoop.hdds.utils.db.RocksDatabase: Deleting sst file /001319.sst corresponding to column family keyTable from db: /var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState/om.db-c2ae0246-626b-4a09-b1c8-0e76997c26ee
...
...
2023-08-15 17:07:18,444 INFO [OM StateMachine ApplyTransaction Thread - 0]-org.apache.hadoop.ozone.om.request.snapshot.OMSnapshotCreateRequest: Created snapshot: 'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073' with snapshotId: '002cfe90-0684-4dd3-9239-487a92067fa6' under path 'vol23io/bucket870io'
...
...
2023-08-15 17:07:41,334 INFO [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Started snap diff report generation for volume: 'vol23io', bucket: 'bucket870io', fromSnapshot: 'cm-46-1692041157584-0', toSnapshot: 'cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073'
2023-08-15 17:07:41,334 INFO [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading snapshot. Table key: /vol23io/bucket870io/cm-46-1692041157584-0
2023-08-15 17:07:41,335 INFO [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,371 INFO [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache: Loading snapshot. Table key: /vol23io/bucket870io/cm-tmp-2651cc03-a0e7-4fba-ab6c-810a863f4073
2023-08-15 17:07:41,371 INFO [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo: OmKeyInfo.getCodec ignorePipeline = true
2023-08-15 17:07:41,438 WARN [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$3(SnapshotDiffManager.java:949)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,439 WARN [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,522 WARN [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Failed to get SST diff file using RocksDBCheckpointDiffer. It will fallback to full diff now.
java.io.FileNotFoundException: Can't find SST file: 001319.sst
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:688)
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:954)
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:938)
at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:875)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:1237)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:1067)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$4(SnapshotDiffManager.java:959)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:1015)
at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$2(SnapshotDiffManager.java:742)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-08-15 17:07:41,522 WARN [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: RocksDBCheckpointDiffer is not available, falling back to slow path
2023-08-15 17:07:41,531 INFO [snapshot-diff-job-thread-id-2]-org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Starting diff report generation for jobId: 2239d0e4-86ff-48f9-9919-08a6b8e5ad7f.
...
{code}
> SST files are missing on optimized snapDiff path.
> -------------------------------------------------
>
> Key: HDDS-8940
> URL: https://issues.apache.org/jira/browse/HDDS-8940
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Hemant Kumar
> Assignee: Hemant Kumar
> Priority: Major
> Attachments: HDDS-8940_Compaction_Dag.png
>
>
> While running snapDiff, we are seeing SST files missing on optimized snapDiff path.
> {code}
> 2023-06-23 19:59:16,323 [snapshot-diff-job-thread-id-14] ERROR org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager: Caught checked exception during diff report generation for volume: volume1 bucket: bucket1, fromSnapshot: alma2 and toSnapshot: cm-tmp-0ae3d532-237d-4df2-83f9-4844d153521e
> java.io.FileNotFoundException: Can't find SST file: 010788.sst
> at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getAbsoluteSstFilePath(RocksDBCheckpointDiffer.java:654)
> at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.filterRelevantSstFilesFullPath(RocksDBCheckpointDiffer.java:949)
> at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:933)
> at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:868)
> at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:929)
> at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFilesAndDiffKeysToObjectIdToKeyMap(SnapshotDiffManager.java:793)
> at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.generateSnapshotDiffReport(SnapshotDiffManager.java:721)
> at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.lambda$0(SnapshotDiffManager.java:565)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org