You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sumitagrawl (via GitHub)" <gi...@apache.org> on 2023/06/01 13:50:55 UTC
[GitHub] [ozone] sumitagrawl opened a new pull request, #4817: HDDS-8742. Quota repair counts of namespace and space usages
sumitagrawl opened a new pull request, #4817:
URL: https://github.com/apache/ozone/pull/4817
## What changes were proposed in this pull request?
- During upgrade, hook is provided to trigger quota repair task
- This task will re-calculate quota usages for space and namespace considering keys, files and directories
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8742
## How was this patch tested?
- Unit test is added and verified
- Upgrade test is done observing, only first time is called during upgrade and count is corrected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226883921
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
+ try {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.acquireReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ repairCount();
+ } finally {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.releaseReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ executor.shutdown();
+ LOG.info("Completed quota repair task");
+ }
+ }
+
+ private void prepareAllVolumeBucketInfo() throws IOException {
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmVolumeArgs>>
+ iterator = metadataManager.getVolumeTable().iterator()) {
+
+ OmVolumeArgs omVolumeArgs;
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmVolumeArgs> entry =
+ iterator.next();
+ omVolumeArgs = entry.getValue();
+ getAllBuckets(omVolumeArgs.getVolume(), omVolumeArgs.getObjectID());
+ }
+ }
+ }
+
+ private void getAllBuckets(String volumeName, long volumeId)
+ throws IOException {
+ List<OmBucketInfo> bucketList = metadataManager.listBuckets(
+ volumeName, null, null, Integer.MAX_VALUE, false);
+ for (OmBucketInfo bucketInfo : bucketList) {
+ bucketInfo.incrUsedNamespace(-bucketInfo.getUsedNamespace());
Review Comment:
Yes, it does reset to "0", reusing interface of incrusedNamespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226937000
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
+ try {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.acquireReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ repairCount();
+ } finally {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.releaseReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ executor.shutdown();
+ LOG.info("Completed quota repair task");
+ }
+ }
+
+ private void prepareAllVolumeBucketInfo() throws IOException {
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmVolumeArgs>>
+ iterator = metadataManager.getVolumeTable().iterator()) {
+
+ OmVolumeArgs omVolumeArgs;
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmVolumeArgs> entry =
+ iterator.next();
+ omVolumeArgs = entry.getValue();
+ getAllBuckets(omVolumeArgs.getVolume(), omVolumeArgs.getObjectID());
+ }
+ }
+ }
+
+ private void getAllBuckets(String volumeName, long volumeId)
+ throws IOException {
+ List<OmBucketInfo> bucketList = metadataManager.listBuckets(
+ volumeName, null, null, Integer.MAX_VALUE, false);
+ for (OmBucketInfo bucketInfo : bucketList) {
+ bucketInfo.incrUsedNamespace(-bucketInfo.getUsedNamespace());
+ bucketInfo.incrUsedBytes(-bucketInfo.getUsedBytes());
+ nameBucketInfoMap.put(buildNamePath(volumeName,
+ bucketInfo.getBucketName()), bucketInfo);
+ idBucketInfoMap.put(buildIdPath(volumeId, bucketInfo.getObjectID()),
+ bucketInfo);
+ }
+ }
+
+ private String buildNamePath(String volumeName, String bucketName) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeName)
+ .append(OM_KEY_PREFIX)
+ .append(bucketName)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private String buildIdPath(long volumeId, long bucketId) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeId)
+ .append(OM_KEY_PREFIX)
+ .append(bucketId)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private void repairCount() throws Exception {
+ LOG.info("Starting quota repair for all keys, files and directories");
+ try {
+ nameBucketInfoMap.keySet().stream().forEach(e -> keyCountMap.put(e,
Review Comment:
this is common function for key, file and directory, it should be context-less. so to move this inside, need pass nameBucketInfoMap/idBucketInfoMap to method. Since the number of bucket is few thousands and memory operation takin less than second, this movement will not provide benefits. So its avoided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 merged pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 merged PR #4817:
URL: https://github.com/apache/ozone/pull/4817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1214085337
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ executor = Executors.newFixedThreadPool(3);
+ prepareAllVolumeBucketInfo();
+ repairCount();
+ LOG.info("Completed quota repair task");
+ executor.shutdown();
Review Comment:
@sumitagrawl Thanks for working on this patch. One small thing, in case of exception, how the executor will be shutdown ? Can we have it in finally block ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1587738646
[HDDS-8824](https://issues.apache.org/jira/browse/HDDS-8824) is created to track CLI tool based option for re-calculate quota.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1587768620
> [HDDS-8824](https://issues.apache.org/jira/browse/HDDS-8824) is created to track CLI tool based option for re-calculate quota.
Can we think of triggering quota repair at bucket level in Recon instead of CLI, will be more intuitive and user friendly as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] krishnaasawa1 commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "krishnaasawa1 (via GitHub)" <gi...@apache.org>.
krishnaasawa1 commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226377009
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3930,4 +3930,15 @@
Max numbers of keys changed allowed for a snapshot diff job.
</description>
</property>
+
+ <property>
+ <name>ozone.om.upgrade.quota.recalculate.enable</name>
Review Comment:
Cosmetic "enabled". All other properties in ozone uses enabled.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
Review Comment:
How the task threads decided as 3 per table type?
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
+ try {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.acquireReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ repairCount();
+ } finally {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.releaseReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ executor.shutdown();
+ LOG.info("Completed quota repair task");
+ }
+ }
+
+ private void prepareAllVolumeBucketInfo() throws IOException {
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmVolumeArgs>>
+ iterator = metadataManager.getVolumeTable().iterator()) {
+
+ OmVolumeArgs omVolumeArgs;
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmVolumeArgs> entry =
+ iterator.next();
+ omVolumeArgs = entry.getValue();
+ getAllBuckets(omVolumeArgs.getVolume(), omVolumeArgs.getObjectID());
+ }
+ }
+ }
+
+ private void getAllBuckets(String volumeName, long volumeId)
+ throws IOException {
+ List<OmBucketInfo> bucketList = metadataManager.listBuckets(
+ volumeName, null, null, Integer.MAX_VALUE, false);
+ for (OmBucketInfo bucketInfo : bucketList) {
+ bucketInfo.incrUsedNamespace(-bucketInfo.getUsedNamespace());
+ bucketInfo.incrUsedBytes(-bucketInfo.getUsedBytes());
+ nameBucketInfoMap.put(buildNamePath(volumeName,
+ bucketInfo.getBucketName()), bucketInfo);
+ idBucketInfoMap.put(buildIdPath(volumeId, bucketInfo.getObjectID()),
+ bucketInfo);
+ }
+ }
+
+ private String buildNamePath(String volumeName, String bucketName) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeName)
+ .append(OM_KEY_PREFIX)
+ .append(bucketName)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private String buildIdPath(long volumeId, long bucketId) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeId)
+ .append(OM_KEY_PREFIX)
+ .append(bucketId)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private void repairCount() throws Exception {
+ LOG.info("Starting quota repair for all keys, files and directories");
+ try {
+ nameBucketInfoMap.keySet().stream().forEach(e -> keyCountMap.put(e,
Review Comment:
Even this can move to recalculateUsages thread context if layout and boolean is passed.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
+ try {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.acquireReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ repairCount();
+ } finally {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.releaseReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ executor.shutdown();
+ LOG.info("Completed quota repair task");
+ }
+ }
+
+ private void prepareAllVolumeBucketInfo() throws IOException {
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmVolumeArgs>>
+ iterator = metadataManager.getVolumeTable().iterator()) {
+
+ OmVolumeArgs omVolumeArgs;
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmVolumeArgs> entry =
+ iterator.next();
+ omVolumeArgs = entry.getValue();
+ getAllBuckets(omVolumeArgs.getVolume(), omVolumeArgs.getObjectID());
+ }
+ }
+ }
+
+ private void getAllBuckets(String volumeName, long volumeId)
+ throws IOException {
+ List<OmBucketInfo> bucketList = metadataManager.listBuckets(
+ volumeName, null, null, Integer.MAX_VALUE, false);
+ for (OmBucketInfo bucketInfo : bucketList) {
+ bucketInfo.incrUsedNamespace(-bucketInfo.getUsedNamespace());
+ bucketInfo.incrUsedBytes(-bucketInfo.getUsedBytes());
+ nameBucketInfoMap.put(buildNamePath(volumeName,
+ bucketInfo.getBucketName()), bucketInfo);
+ idBucketInfoMap.put(buildIdPath(volumeId, bucketInfo.getObjectID()),
+ bucketInfo);
+ }
+ }
+
+ private String buildNamePath(String volumeName, String bucketName) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeName)
+ .append(OM_KEY_PREFIX)
+ .append(bucketName)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private String buildIdPath(long volumeId, long bucketId) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeId)
+ .append(OM_KEY_PREFIX)
+ .append(bucketId)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private void repairCount() throws Exception {
+ LOG.info("Starting quota repair for all keys, files and directories");
+ try {
+ nameBucketInfoMap.keySet().stream().forEach(e -> keyCountMap.put(e,
+ new CountPair()));
+ idBucketInfoMap.keySet().stream().forEach(e -> fileCountMap.put(e,
+ new CountPair()));
+ idBucketInfoMap.keySet().stream().forEach(e -> directoryCountMap.put(e,
+ new CountPair()));
+
+ List<Future<?>> tasks = new ArrayList<>();
+ tasks.add(executor.submit(() -> recalculateUsages(
Review Comment:
Instead of passing multiple params in recalculateUsages, can't we pass just pass Layout and avoid passing other params. For DirectoryTable yes we need to take care may be the boolean passing apart from layout will take care.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
Review Comment:
How we decided on batch count value as 5K?
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
+ try {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.acquireReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ repairCount();
+ } finally {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.releaseReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ executor.shutdown();
+ LOG.info("Completed quota repair task");
+ }
+ }
+
+ private void prepareAllVolumeBucketInfo() throws IOException {
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmVolumeArgs>>
+ iterator = metadataManager.getVolumeTable().iterator()) {
+
+ OmVolumeArgs omVolumeArgs;
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmVolumeArgs> entry =
+ iterator.next();
+ omVolumeArgs = entry.getValue();
+ getAllBuckets(omVolumeArgs.getVolume(), omVolumeArgs.getObjectID());
+ }
+ }
+ }
+
+ private void getAllBuckets(String volumeName, long volumeId)
+ throws IOException {
+ List<OmBucketInfo> bucketList = metadataManager.listBuckets(
+ volumeName, null, null, Integer.MAX_VALUE, false);
+ for (OmBucketInfo bucketInfo : bucketList) {
+ bucketInfo.incrUsedNamespace(-bucketInfo.getUsedNamespace());
Review Comment:
I suppose this is to reset the values to 0 and later capturecount is repopulating correct new values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226886885
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
Review Comment:
Test is performed with varying number of threads, making 4, 5 and also reducing to 2, 1. But with thread count "3" provided best result. Reason is,
- higher number of thread increases thread contention
- time taken by decode task is approx 3 times the retrival task as time measured
So based on this result, it found to be "3" as number of task per table type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226894936
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
Review Comment:
Test is performed with increasing the batch size, can not distinguish much performance improvement, may be 2-5 second with 50 milliion records, but due to variation with each run, can not clearly identify any improvement.
Also considered memory usages with increasing batch size (for key / file):
5K (records) * 5 (queue size -3 / threads - 2) * 2 (table types) * 10KB (data size of decoded value) ~= 500MB + addition for directory with no decoding of directory data.
So this value is considered keeping balance between batch size and memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1590295974
> > Can we think of triggering quota repair at bucket level in Recon instead of CLI, will be more intuitive and user friendly as well
>
> How would OM learn about the repaired values? Right now information flows from OM to Recon, not the other way around. The proposed CLI would be run on each OM host, and would be provided the OM DB to operate on so it can run offline while OM is shut down.
@errose28 thanks for clarifying, however in future we want to make Recon a tool which can trigger actions other way around as well, so if we can provide om db file path to recon while quota repair, would it make sense and push back the file to OM db file same path after repair ? Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1596530321
Thanks @sumitagrawl for the contribution & @errose28 , @devmadhuu , @krishnaasawa1 for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1572108944
@sumitagrawl CI [is run](https://github.com/sumitagrawl/ozone/actions?query=branch%3AHDDS-8742) in forks for each push. There is some limit on the concurrent Github Actions jobs/workflows for each repo (although I don't know the details). If you push in multiple batches, you might want to cancel runs for outdated versions of your code, otherwise the latest commits will be queued for a long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1231809947
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/QuotaRepairUpgradeAction.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.service.QuotaRepairTask;
+
+import static org.apache.hadoop.ozone.om.upgrade.OMLayoutFeature.QUOTA;
+import static org.apache.hadoop.ozone.upgrade.LayoutFeature.UpgradeActionType.ON_FIRST_UPGRADE_START;
+
+/**
+ * Quota repair for usages action to be triggered during first upgrade.
+ */
+@UpgradeActionOm(type = ON_FIRST_UPGRADE_START, feature =
+ QUOTA)
+public class QuotaRepairUpgradeAction implements OmUpgradeAction {
+ @Override
+ public void execute(OzoneManager arg) throws Exception {
Review Comment:
Does this take place in prepare mode? Or are writes enabled during the the time this task runs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1231964697
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/QuotaRepairUpgradeAction.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.upgrade;
+
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.service.QuotaRepairTask;
+
+import static org.apache.hadoop.ozone.om.upgrade.OMLayoutFeature.QUOTA;
+import static org.apache.hadoop.ozone.upgrade.LayoutFeature.UpgradeActionType.ON_FIRST_UPGRADE_START;
+
+/**
+ * Quota repair for usages action to be triggered during first upgrade.
+ */
+@UpgradeActionOm(type = ON_FIRST_UPGRADE_START, feature =
+ QUOTA)
+public class QuotaRepairUpgradeAction implements OmUpgradeAction {
+ @Override
+ public void execute(OzoneManager arg) throws Exception {
Review Comment:
Yes, Its triggered in prepare mode and takes read lock for the duration, so no write is allowed. Only read is allowed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1590163582
> Can we think of triggering quota repair at bucket level in Recon instead of CLI, will be more intuitive and user friendly as well
How would OM learn about the repaired values? Right now information flows from OM to Recon, not the other way around. The proposed CLI would be run on each OM host, and would be provided the OM DB to operate on so it can run offline while OM is shut down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #4817:
URL: https://github.com/apache/ozone/pull/4817#issuecomment-1590795787
> > > Can we think of triggering quota repair at bucket level in Recon instead of CLI, will be more intuitive and user friendly as well
> >
> >
> > How would OM learn about the repaired values? Right now information flows from OM to Recon, not the other way around. The proposed CLI would be run on each OM host, and would be provided the OM DB to operate on so it can run offline while OM is shut down.
>
> @errose28 thanks for clarifying, however in future we want to make Recon a tool which can trigger actions other way around as well, so if we can provide om db file path to recon while quota repair, would it make sense and push back the file to OM db file same path after repair ? Am I missing something?
@devmadhuu There are below reason where this kind of operation normally is avoided, even from recon,
1. If customer have 1 billion keys, it will take 30 min as per current perf measurement for each bucket. It will lock the bucket for the duration and no write operation will be allowed.
2. This will be debug operation if count goes wrong where it make take this much time
3. If Recon integration is required, may need think another solution then CLI trigger from Recon as part of this.
This can be discussed as part of [HDDS-8824](https://issues.apache.org/jira/browse/HDDS-8824) for CLI cases and further support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226933418
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/QuotaRepairTask.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.ozone.om.service;
+
+import com.google.common.util.concurrent.UncheckedExecutionException;
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.hadoop.hdds.utils.db.BatchOperation;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.hadoop.ozone.OzoneConsts.OM_KEY_PREFIX;
+import static org.apache.hadoop.ozone.om.lock.OzoneManagerLock.Resource.BUCKET_LOCK;
+
+/**
+ * Quota repair task.
+ */
+public class QuotaRepairTask {
+ private static final Logger LOG = LoggerFactory.getLogger(
+ QuotaRepairTask.class);
+ private static final int BATCH_SIZE = 5000;
+ private static final int TASK_THREAD_CNT = 3;
+ private final OMMetadataManager metadataManager;
+ private final Map<String, OmBucketInfo> nameBucketInfoMap = new HashMap<>();
+ private final Map<String, OmBucketInfo> idBucketInfoMap = new HashMap<>();
+ private ExecutorService executor;
+ private final Map<String, CountPair> keyCountMap = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> fileCountMap
+ = new ConcurrentHashMap<>();
+ private final Map<String, CountPair> directoryCountMap
+ = new ConcurrentHashMap<>();
+
+ public QuotaRepairTask(OMMetadataManager metadataManager) {
+ this.metadataManager = metadataManager;
+ }
+
+ public void repair() throws Exception {
+ LOG.info("Starting quota repair task");
+ prepareAllVolumeBucketInfo();
+
+ IOzoneManagerLock lock = metadataManager.getLock();
+ // thread pool with 3 Table type * (1 task each + 3 thread each)
+ executor = Executors.newFixedThreadPool(12);
+ try {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.acquireReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ repairCount();
+ } finally {
+ nameBucketInfoMap.values().stream().forEach(e -> lock.releaseReadLock(
+ BUCKET_LOCK, e.getVolumeName(), e.getBucketName()));
+ executor.shutdown();
+ LOG.info("Completed quota repair task");
+ }
+ }
+
+ private void prepareAllVolumeBucketInfo() throws IOException {
+ try (TableIterator<String, ? extends Table.KeyValue<String, OmVolumeArgs>>
+ iterator = metadataManager.getVolumeTable().iterator()) {
+
+ OmVolumeArgs omVolumeArgs;
+ while (iterator.hasNext()) {
+ Table.KeyValue<String, OmVolumeArgs> entry =
+ iterator.next();
+ omVolumeArgs = entry.getValue();
+ getAllBuckets(omVolumeArgs.getVolume(), omVolumeArgs.getObjectID());
+ }
+ }
+ }
+
+ private void getAllBuckets(String volumeName, long volumeId)
+ throws IOException {
+ List<OmBucketInfo> bucketList = metadataManager.listBuckets(
+ volumeName, null, null, Integer.MAX_VALUE, false);
+ for (OmBucketInfo bucketInfo : bucketList) {
+ bucketInfo.incrUsedNamespace(-bucketInfo.getUsedNamespace());
+ bucketInfo.incrUsedBytes(-bucketInfo.getUsedBytes());
+ nameBucketInfoMap.put(buildNamePath(volumeName,
+ bucketInfo.getBucketName()), bucketInfo);
+ idBucketInfoMap.put(buildIdPath(volumeId, bucketInfo.getObjectID()),
+ bucketInfo);
+ }
+ }
+
+ private String buildNamePath(String volumeName, String bucketName) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeName)
+ .append(OM_KEY_PREFIX)
+ .append(bucketName)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private String buildIdPath(long volumeId, long bucketId) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append(OM_KEY_PREFIX)
+ .append(volumeId)
+ .append(OM_KEY_PREFIX)
+ .append(bucketId)
+ .append(OM_KEY_PREFIX);
+ return builder.toString();
+ }
+
+ private void repairCount() throws Exception {
+ LOG.info("Starting quota repair for all keys, files and directories");
+ try {
+ nameBucketInfoMap.keySet().stream().forEach(e -> keyCountMap.put(e,
+ new CountPair()));
+ idBucketInfoMap.keySet().stream().forEach(e -> fileCountMap.put(e,
+ new CountPair()));
+ idBucketInfoMap.keySet().stream().forEach(e -> directoryCountMap.put(e,
+ new CountPair()));
+
+ List<Future<?>> tasks = new ArrayList<>();
+ tasks.add(executor.submit(() -> recalculateUsages(
Review Comment:
this is common function for key, file and directory, it should be context-less, or else if-else will come inside the function increasing its complexity. So required parameter is passed avoiding if-else for 3 types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4817: HDDS-8742. Quota repair counts of namespace and space usages
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4817:
URL: https://github.com/apache/ozone/pull/4817#discussion_r1226965754
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3930,4 +3930,15 @@
Max numbers of keys changed allowed for a snapshot diff job.
</description>
</property>
+
+ <property>
+ <name>ozone.om.upgrade.quota.recalculate.enable</name>
Review Comment:
Updated...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org