You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/09/21 16:48:11 UTC
[GitHub] [ozone] duongkame opened a new pull request, #3771: HDDS-7229: Introduce container location cache in ScmClient
duongkame opened a new pull request, #3771:
URL: https://github.com/apache/ozone/pull/3771
## What changes were proposed in this pull request?
Introduce container location cache in ScmClient.
https://issues.apache.org/jira/browse/HDDS-7229
## How was this patch tested?
Standard CI with added unit test to cover new ScmClient interfaces: https://github.com/duongkame/ozone/actions/runs/3099408939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3771:
URL: https://github.com/apache/ozone/pull/3771#issuecomment-1254219618
Looks good, will continue to review it. For now I think a merge conflict needs to be resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3771:
URL: https://github.com/apache/ozone/pull/3771#discussion_r984139661
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -52,4 +107,40 @@ public void setUpdateServiceGrpcClient(
public SCMUpdateServiceGrpcClient getUpdateServiceGrpcClient() {
return updateServiceGrpcClient;
}
+
+ public Pipeline getContainerLocation(long containerId, boolean forceRefresh)
+ throws IOException {
+ if (forceRefresh) {
+ containerLocationCache.invalidate(containerId);
+ }
+ try {
+ return containerLocationCache.get(containerId);
+ } catch (ExecutionException e) {
+ return handleCacheExecutionException(e);
+ }
+ }
+
+ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
Review Comment:
A key is designed to have multiple blocks and thus multiple containers. Indeed, we always use this interface and don't really need `getContainerLocation`. Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime merged pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime merged PR #3771:
URL: https://github.com/apache/ozone/pull/3771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3771:
URL: https://github.com/apache/ozone/pull/3771#discussion_r983183643
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestScmClient.java:
##########
@@ -0,0 +1,262 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.client.ReplicationType;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.container.ContainerInfo;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol;
+import org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Stream;
+
+import static com.google.common.collect.Sets.newHashSet;
+import static java.util.Arrays.asList;
+import static org.apache.commons.lang.RandomStringUtils.randomAlphabetic;
+import static org.apache.hadoop.hdds.client.ReplicationConfig.fromTypeAndFactor;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * ScmClient test-cases.
+ */
+public class TestScmClient {
+ private ScmBlockLocationProtocol scmBlockLocationProtocol;
+ private StorageContainerLocationProtocol containerLocationProtocol;
+ private OzoneConfiguration conf;
+ private ScmClient scmClient;
+
+ @BeforeEach
+ public void setUp() {
+ scmBlockLocationProtocol = mock(ScmBlockLocationProtocol.class);
+ containerLocationProtocol = mock(StorageContainerLocationProtocol.class);
+ conf = new OzoneConfiguration();
+ scmClient = new ScmClient(scmBlockLocationProtocol,
+ containerLocationProtocol, conf);
+ }
+
+ private static Stream<Arguments> getContainerLocationTestCases() {
+ return Stream.of(
+ Arguments.of("New key",
+ newHashSet(1L, 2L), 3L, false, 1),
+
+ Arguments.of("Existing key",
+ newHashSet(1L, 2L), 1L, false, 1),
+
+ Arguments.of("New key with force refresh",
+ newHashSet(1L, 2L), 3L, true, 1),
+
+ Arguments.of("Existing key, force refresh",
+ newHashSet(1L, 2L), 1L, true, 2)
+ );
+ }
+
+ @ParameterizedTest
+ @MethodSource("getContainerLocationTestCases")
+ public void testGetContainerLocation(String testCases,
+ Set<Long> prepopulatedIds,
+ long testId,
Review Comment:
```suggestion
long containerIdToTest,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3771:
URL: https://github.com/apache/ozone/pull/3771#issuecomment-1254221053
cc @tanvipenumudy @DaveTeng0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3771:
URL: https://github.com/apache/ozone/pull/3771#issuecomment-1255524111
CC @neils-dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3771:
URL: https://github.com/apache/ozone/pull/3771#issuecomment-1261900812
I plan to go over the test code one more time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3771:
URL: https://github.com/apache/ozone/pull/3771#discussion_r984931010
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestScmClient.java:
##########
@@ -0,0 +1,191 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.hdds.client.ReplicationFactor;
+import org.apache.hadoop.hdds.client.ReplicationType;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.apache.hadoop.hdds.scm.container.ContainerInfo;
+import org.apache.hadoop.hdds.scm.container.common.helpers.ContainerWithPipeline;
+import org.apache.hadoop.hdds.scm.pipeline.Pipeline;
+import org.apache.hadoop.hdds.scm.pipeline.PipelineID;
+import org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol;
+import org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Stream;
+
+import static com.google.common.collect.Sets.newHashSet;
+import static java.util.Arrays.asList;
+import static org.apache.commons.lang.RandomStringUtils.randomAlphabetic;
+import static org.apache.hadoop.hdds.client.ReplicationConfig.fromTypeAndFactor;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+/**
+ * ScmClient test-cases.
+ */
+public class TestScmClient {
+ private ScmBlockLocationProtocol scmBlockLocationProtocol;
+ private StorageContainerLocationProtocol containerLocationProtocol;
+ private OzoneConfiguration conf;
+ private ScmClient scmClient;
+
+ @BeforeEach
+ public void setUp() {
+ scmBlockLocationProtocol = mock(ScmBlockLocationProtocol.class);
+ containerLocationProtocol = mock(StorageContainerLocationProtocol.class);
+ conf = new OzoneConfiguration();
+ scmClient = new ScmClient(scmBlockLocationProtocol,
+ containerLocationProtocol, conf);
+ }
+
+ private static Stream<Arguments> getContainerLocationsTestCases() {
+ return Stream.of(
+ Arguments.of("Existing keys",
+ newHashSet(1L, 2L, 3L), newHashSet(2L, 3L), false, newHashSet()),
+
+ Arguments.of("New keys",
+ newHashSet(1L, 2L), newHashSet(3L, 4L),
+ false, newHashSet(3L, 4L)),
+
+ Arguments.of("Partial new keys",
+ newHashSet(1L, 2L), newHashSet(1L, 3L, 4L),
+ false, newHashSet(3L, 4L)),
+
+ Arguments.of("Existing keys with force refresh",
+ newHashSet(1L, 2L, 3L), newHashSet(2L, 3L),
+ true, newHashSet(2L, 3L)),
+
+ Arguments.of("New keys with force refresh",
+ newHashSet(1L, 2L), newHashSet(3L, 4L),
+ true, newHashSet(3L, 4L)),
+
+ Arguments.of("Partial new keys with force refresh",
+ newHashSet(1L, 2L), newHashSet(1L, 3L, 4L),
+ true, newHashSet(1L, 3L, 4L))
+ );
+ }
+ @ParameterizedTest
+ @MethodSource("getContainerLocationsTestCases")
+ public void testGetContainerLocations(String testCases,
Review Comment:
```suggestion
public void testGetContainerLocations(String testCaseName,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
duongkame commented on PR #3771:
URL: https://github.com/apache/ozone/pull/3771#issuecomment-1254377070
> Looks good, will continue to review it. For now I think a merge conflict needs to be resolved.
Thanks @kerneltime . I resolved the merge conflict.
Also, this is done without DatanodeDetails normalization. Just did an experiment to confirm the cache size on top of the changes in this PR and the results are documented as per (HDDS-7249)[https://issues.apache.org/jira/browse/HDDS-7249]. In summary, 10K containers take 35mb heap and that number linearly grows with the number of containers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3771: HDDS-7229. Introduce container location cache in ScmClient
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3771:
URL: https://github.com/apache/ozone/pull/3771#discussion_r983178750
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -52,4 +107,40 @@ public void setUpdateServiceGrpcClient(
public SCMUpdateServiceGrpcClient getUpdateServiceGrpcClient() {
return updateServiceGrpcClient;
}
+
+ public Pipeline getContainerLocation(long containerId, boolean forceRefresh)
+ throws IOException {
+ if (forceRefresh) {
+ containerLocationCache.invalidate(containerId);
+ }
+ try {
+ return containerLocationCache.get(containerId);
+ } catch (ExecutionException e) {
+ return handleCacheExecutionException(e);
+ }
+ }
+
+ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
Review Comment:
Is this for objects with multiple containers? Why have `getContainerLocation`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org