You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "hemantk-12 (via GitHub)" <gi...@apache.org> on 2023/02/03 21:27:43 UTC
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4236: HDDS-7871. Fix false positive in KeyManagerImpl#createFakeDirIfShould()
hemantk-12 commented on code in PR #4236:
URL: https://github.com/apache/ozone/pull/4236#discussion_r1096286520
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1351,6 +1353,53 @@ public void testGetFileStatusWithFakeDir() throws IOException {
Assert.assertTrue(ozoneFileStatus.isFile());
}
+ @Test
+ public void testGetFileStatusWithFakeDirFalsePositive() throws IOException {
+ String dirName = "foo2";
+ String fileName = "bar2";
+ String keyName1 = "foo1";
+ // keyName2 = "foo2/bar2"
+ String keyName2 = dirName + OZONE_URI_DELIMITER + fileName;
+ OzoneFileStatus ozoneFileStatus;
+
+ // create a key "foo1" in bucket1
+ OmKeyArgs keyArgs = createBuilder(BUCKET_NAME).setKeyName(keyName1).build();
+ OpenKeySession keySession = writeClient.openKey(keyArgs);
+ keyArgs.setLocationInfoList(
+ keySession.getKeyInfo().getLatestVersionLocations().getLocationList());
+ writeClient.commitKey(keyArgs, keySession.getId());
+
+ // create a key "foo2/bar2" in bucket2
+ keyArgs = createBuilder(BUCKET2_NAME).setKeyName(keyName2).build();
+ keySession = writeClient.createFile(keyArgs, true, true);
+ keyArgs.setLocationInfoList(
+ keySession.getKeyInfo().getLatestVersionLocations().getLocationList());
+ writeClient.commitKey(keyArgs, keySession.getId());
Review Comment:
nit: These lines (1373-1377) are repetitive as line from 1366-1370. May be create a helper function or use existing one if there is any.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java:
##########
@@ -1178,21 +1178,18 @@ private OmKeyInfo createFakeDirIfShould(String volume, String bucket,
String keyName, BucketLayout layout) throws IOException {
OmKeyInfo fakeDirKeyInfo = null;
String dirKey = OzoneFSUtils.addTrailingSlashIfNeeded(keyName);
- String fileKeyBytes = metadataManager.getOzoneKey(volume, bucket, keyName);
+ String targetKey = OzoneFSUtils.addTrailingSlashIfNeeded(
+ metadataManager.getOzoneKey(volume, bucket, keyName));
try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>>
keyTblItr = metadataManager.getKeyTable(layout).iterator()) {
- Table.KeyValue<String, OmKeyInfo> keyValue =
- keyTblItr
- .seek(OzoneFSUtils.addTrailingSlashIfNeeded(fileKeyBytes));
-
- if (keyValue != null) {
- Path fullPath = Paths.get(keyValue.getValue().getKeyName());
- Path subPath = Paths.get(dirKey);
- OmKeyInfo omKeyInfo = keyValue.getValue();
- if (fullPath.startsWith(subPath)) {
- // create fake directory
- fakeDirKeyInfo = createDirectoryKey(omKeyInfo, dirKey);
- }
+ Table.KeyValue<String, OmKeyInfo> keyValue = keyTblItr.seek(targetKey);
+
+ // HDDS-7871: RocksIterator#seek() may position at the key
Review Comment:
Is it because seek may position past target or bloom filter or when item is not in the table?
As per my understanding it seems like [false positive because of bloom filter](https://github.com/facebook/rocksdb/wiki/Prefix-Seek). Please correct me if I'm wrong.
When I was testing it, my test was failure for following scenario.
Items in the table was:
| KeyInfoTable |
|-------------------|
| /vol1/bucket1/key1 |
| /vol1/bucket1/key2 |
| /vol1/bucket1/key3 |
| /vol1/bucket1/key4 |
| /vol1/bucket1/key5 |
Was looking for `/vol2/bucket2/key1` and getting false positive.
May point is if it is because of bloom filter and seek is pointing to wrong item, we might miss the "fakeDir".
Eg. You have following table
| KeyInfoTable |
|-----------------------|
| /vol1/bucket1/dir1/key1 |
| /vol1/bucket1/dir1/key2 |
| /vol1/bucket1/dir1/key3 |
| /vol1/bucket2/dir1/key1 |
| /vol1/bucket2/dir1/key2 |
and now you are looking for `/vol1/bucket2/dir1` but seek is pointing to ` /vol1/bucket1/dir1/key1`. So it will be a miss all the time now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org