You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/10/08 12:31:38 UTC
[GitHub] [ozone] aryangupta1998 opened a new pull request #2724: HDDS-5659. Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1
aryangupta1998 opened a new pull request #2724:
URL: https://github.com/apache/ozone/pull/2724
## What changes were proposed in this pull request?
In OmMetadataManagerImpl we have a function getKeyTable() which uses the 'isBucketFSOptimized' flag to determine FSO buckets, with help of this Jira we will be independent of this flag and deduce bucketLayout directly from OmBucketInfo.
Divided the task into two parts. This task is part-1, where we will modify the keys/files related to FSO and OBS classes.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5659
## How was this patch tested?
Tested Manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on a change in pull request #2724: HDDS-5659. Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #2724:
URL: https://github.com/apache/ozone/pull/2724#discussion_r730709461
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMClientResponse.java
##########
@@ -46,6 +52,21 @@ public BucketLayout getBucketLayout() {
return BucketLayout.DEFAULT;
}
+ protected BucketLayout getBucketLayout(OMMetadataManager omMetadataManager,
+ String volName, String buckName) {
+ if (omMetadataManager == null) {
+ return BucketLayout.DEFAULT;
+ }
+ String buckKey = omMetadataManager.getBucketKey(volName, buckName);
+ try {
+ OmBucketInfo buckInfo = omMetadataManager.getBucketTable().get(buckKey);
+ return buckInfo.getBucketLayout();
+ } catch (IOException e) {
+ LOG.error("Cannot find the key: " + buckKey);
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a change in pull request #2724: HDDS-5659. Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #2724:
URL: https://github.com/apache/ozone/pull/2724#discussion_r730698573
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java
##########
@@ -30,10 +30,7 @@
import com.google.common.base.Preconditions;
import org.apache.hadoop.hdds.client.ReplicationConfig;
import org.apache.hadoop.ozone.OzoneAcl;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
-import org.apache.hadoop.ozone.om.helpers.OzoneAclUtil;
-import org.apache.hadoop.ozone.om.helpers.OzoneFSUtils;
+import org.apache.hadoop.ozone.om.helpers.*;
Review comment:
@aryangupta1998 Please revert `import org.apache.hadoop.ozone.om.helpers.*;` to specific import statements, this would avoid including everything.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMClientResponse.java
##########
@@ -46,6 +52,21 @@ public BucketLayout getBucketLayout() {
return BucketLayout.DEFAULT;
}
+ protected BucketLayout getBucketLayout(OMMetadataManager omMetadataManager,
+ String volName, String buckName) {
+ if (omMetadataManager == null) {
+ return BucketLayout.DEFAULT;
+ }
+ String buckKey = omMetadataManager.getBucketKey(volName, buckName);
+ try {
+ OmBucketInfo buckInfo = omMetadataManager.getBucketTable().get(buckKey);
+ return buckInfo.getBucketLayout();
+ } catch (IOException e) {
+ LOG.error("Cannot find the key: " + buckKey);
Review comment:
@aryangupta1998 can you please include e object into the log.
`LOG.error("Cannot find the key: " + buckKey, e);`
##########
File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java
##########
@@ -44,12 +44,7 @@
import org.apache.hadoop.hdds.utils.db.Table;
import org.apache.hadoop.ozone.om.OMMetadataManager;
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
-import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
-import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
-import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.*;
Review comment:
@aryangupta1998 Please revert `import org.apache.hadoop.ozone.om.helpers.*;` to specific import statements, this would avoid including everything.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##########
@@ -488,11 +496,28 @@ public static void addFileTableCacheEntry(
omFileInfo.setKeyName(fileName);
omFileInfo.setFileName(fileName);
- omMetadataManager.getKeyTable().addCacheEntry(
- new CacheKey<>(dbFileKey),
+ omMetadataManager.getKeyTable(
+ getBucketLayout(omMetadataManager, omFileInfo.getVolumeName(),
+ omFileInfo.getBucketName()))
+ .addCacheEntry(new CacheKey<>(dbFileKey),
new CacheValue<>(Optional.of(omFileInfo), trxnLogIndex));
}
+ public static BucketLayout getBucketLayout(
+ OMMetadataManager omMetadataManager, String volName, String buckName) {
+ if (omMetadataManager == null) {
+ return BucketLayout.DEFAULT;
+ }
+ String buckKey = omMetadataManager.getBucketKey(volName, buckName);
+ try {
+ OmBucketInfo buckInfo = omMetadataManager.getBucketTable().get(buckKey);
+ return buckInfo.getBucketLayout();
+ } catch (IOException e) {
+ LOG.error("Cannot find the key: " + buckKey);
Review comment:
@aryangupta1998 can you please include e object into the log.
LOG.error("Cannot find the key: " + buckKey, e);
##########
File path: hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java
##########
@@ -291,6 +291,10 @@ boolean recoverTrash(String volumeName, String bucketName,
*
* @return KeyTable.
*/
+ //Table<String, OmKeyInfo> getKeyTable();
Review comment:
@aryangupta1998 please remove this commented statement.
##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequestWithFSO.java
##########
@@ -34,10 +34,7 @@
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
import org.apache.hadoop.ozone.om.OzoneManager;
import org.apache.hadoop.ozone.om.ResolvedBucket;
-import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
-import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.OzoneFSUtils;
+import org.apache.hadoop.ozone.om.helpers.*;
Review comment:
@aryangupta1998 Please revert `import org.apache.hadoop.ozone.om.helpers.*;` to specific import statements, this would avoid including everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr merged pull request #2724: HDDS-5659. Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1
Posted by GitBox <gi...@apache.org>.
rakeshadr merged pull request #2724:
URL: https://github.com/apache/ozone/pull/2724
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] aryangupta1998 commented on a change in pull request #2724: HDDS-5659. Use bucket layout specific DB table in OmMetadataManager.getKeyTable() - part1
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #2724:
URL: https://github.com/apache/ozone/pull/2724#discussion_r730709608
##########
File path: hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java
##########
@@ -291,6 +291,10 @@ boolean recoverTrash(String volumeName, String bucketName,
*
* @return KeyTable.
*/
+ //Table<String, OmKeyInfo> getKeyTable();
Review comment:
Done
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##########
@@ -488,11 +496,28 @@ public static void addFileTableCacheEntry(
omFileInfo.setKeyName(fileName);
omFileInfo.setFileName(fileName);
- omMetadataManager.getKeyTable().addCacheEntry(
- new CacheKey<>(dbFileKey),
+ omMetadataManager.getKeyTable(
+ getBucketLayout(omMetadataManager, omFileInfo.getVolumeName(),
+ omFileInfo.getBucketName()))
+ .addCacheEntry(new CacheKey<>(dbFileKey),
new CacheValue<>(Optional.of(omFileInfo), trxnLogIndex));
}
+ public static BucketLayout getBucketLayout(
+ OMMetadataManager omMetadataManager, String volName, String buckName) {
+ if (omMetadataManager == null) {
+ return BucketLayout.DEFAULT;
+ }
+ String buckKey = omMetadataManager.getBucketKey(volName, buckName);
+ try {
+ OmBucketInfo buckInfo = omMetadataManager.getBucketTable().get(buckKey);
+ return buckInfo.getBucketLayout();
+ } catch (IOException e) {
+ LOG.error("Cannot find the key: " + buckKey);
Review comment:
Done
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java
##########
@@ -30,10 +30,7 @@
import com.google.common.base.Preconditions;
import org.apache.hadoop.hdds.client.ReplicationConfig;
import org.apache.hadoop.ozone.OzoneAcl;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
-import org.apache.hadoop.ozone.om.helpers.OzoneAclUtil;
-import org.apache.hadoop.ozone.om.helpers.OzoneFSUtils;
+import org.apache.hadoop.ozone.om.helpers.*;
Review comment:
Done
##########
File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/file/TestOMDirectoryCreateRequestWithFSO.java
##########
@@ -34,10 +34,7 @@
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
import org.apache.hadoop.ozone.om.OzoneManager;
import org.apache.hadoop.ozone.om.ResolvedBucket;
-import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
-import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.OzoneFSUtils;
+import org.apache.hadoop.ozone.om.helpers.*;
Review comment:
Done
##########
File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java
##########
@@ -44,12 +44,7 @@
import org.apache.hadoop.hdds.utils.db.Table;
import org.apache.hadoop.ozone.om.OMMetadataManager;
import org.apache.hadoop.ozone.om.OmMetadataManagerImpl;
-import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
-import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo;
-import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup;
-import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.helpers.*;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org