[jira] [Created] (HDDS-7913) 'Graph traversal level exceeded allowed maximum' when the difference b/w snapshots is more than 3.

["Sadanand Shenoy (Jira)" <ji...@apache.org>]

Sadanand Shenoy created HDDS-7913:
-------------------------------------

             Summary: 'Graph traversal level exceeded allowed maximum' when the difference b/w snapshots is more than 3.
                 Key: HDDS-7913
                 URL: https://issues.apache.org/jira/browse/HDDS-7913
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Sadanand Shenoy


Getting the below error when calling snapdiff b/w snapshots which are more than 2-3 snapshots apart.
{code:java}
2023-02-07 10:50:07,464 [IPC Server handler 13 on default port 60192] ERROR rocksdiff.RocksDBCheckpointDiffer (RocksDBCheckpointDiffer.java:internalGetSSTDiffList(899)) - Graph traversal level exceeded allowed maximum (1000000). This could be due to invalid input generating a loop in the traversal path. Same SSTs found so far: [000053], different SSTs: []
2023-02-07 10:50:07,465 [IPC Server handler 13 on default port 60192] WARN  ipc.Server (Server.java:logException(3035)) - IPC Server handler 13 on default port 60192, call Call#102 Retry#159 org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest from 127.0.0.1:60273
java.lang.RuntimeException: Graph traversal level exceeded allowed maximum (1000000). This could be due to invalid input generating a loop in the traversal path. Same SSTs found so far: [000053], different SSTs: []
    at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.internalGetSSTDiffList(RocksDBCheckpointDiffer.java:904)
    at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffList(RocksDBCheckpointDiffer.java:788)
    at org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer.getSSTDiffListWithFullPath(RocksDBCheckpointDiffer.java:756)
    at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getDeltaFiles(SnapshotDiffManager.java:231)
    at org.apache.hadoop.ozone.om.snapshot.SnapshotDiffManager.getSnapshotDiffReport(SnapshotDiffManager.java:134)
    at org.apache.hadoop.ozone.om.OmSnapshotManager.getSnapshotDiffReport(OmSnapshotManager.java:250)
    at org.apache.hadoop.ozone.om.OzoneManager.snapshotDiff(OzoneManager.java:4325)
    at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.snapshotDiff(OzoneManagerRequestHandler.java:1216)
    at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:298)
    at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:223)
    at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:177)
    at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
    at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
    at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
    at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) {code}
Unit test to repro 
{code:java}
@Test
public void testSnapDiffWithMultipleSSTs()
    throws IOException, InterruptedException, TimeoutException {
  String volumeName1 = "vol-" + RandomStringUtils.randomNumeric(5);
  String bucketName1 = "buck1";
  String bucketName2 = "bucketz";
  store.createVolume(volumeName1);
  OzoneVolume volume1 = store.getVolume(volumeName1);
  volume1.createBucket(bucketName1);
  volume1.createBucket(bucketName2);
  OzoneBucket bucket1 = volume1.getBucket(bucketName1);
  OzoneBucket bucket2 = volume1.getBucket(bucketName2);
  String keyPrefix = "key-";
  createFileKey(bucket1, keyPrefix);
  String snap1 = "snap" + RandomStringUtils.randomNumeric(5);
  createSnapshot(volumeName1, bucketName1, snap1); // 1.sst
  createFileKey(bucket2, keyPrefix);
  String snap2 = "snap" + RandomStringUtils.randomNumeric(5);
  createSnapshot(volumeName1, bucketName1, snap2); // 1.sst
  createFileKey(bucket2, keyPrefix);
  String snap3 = "snap" + RandomStringUtils.randomNumeric(5);
  createSnapshot(volumeName1, bucketName1, snap3); // 1.sst
  createFileKey(bucket1,keyPrefix);
  String snap4 = "snap" + RandomStringUtils.randomNumeric(5);
  createSnapshot(volumeName1, bucketName1, snap4); // 1.sst 2.sst 3.sst 4.sst
  SnapshotDiffReport diff1 =
      store.snapshotDiff(volumeName1, bucketName1, snap1, snap4);
  Assert.assertEquals(2, diff1.getDiffList().size());
}
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org