You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ashishkumar50 (via GitHub)" <gi...@apache.org> on 2024/02/07 13:05:03 UTC
[PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
ashishkumar50 opened a new pull request, #6189:
URL: https://github.com/apache/ozone/pull/6189
## What changes were proposed in this pull request?
When SCM is in safe mode during restart(or rolling restart), block allocation fails. Instead of failing operation immediately, OM can retry few times(5 times every 3 seconds) to get block info from SCM before failing the operation.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10256
## How was this patch tested?
Integration test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1483973646
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java:
##########
@@ -334,4 +343,46 @@ public void testSCMSafeModeDisabled() throws Exception {
cluster.restartStorageContainerManager(true);
assertFalse(scm.isInSafeMode());
}
+
+ @Test
+ public void testCreateRetryWhileSCMSafeMode() throws Exception {
+ // Test1: Test safe mode when there are no containers in system.
+ cluster.stop();
+
+ try {
+ cluster = builder.build();
+ } catch (IOException e) {
+ fail("Cluster startup failed.");
+ }
+
+ final String rootPath = String.format("%s://%s/",
+ OZONE_OFS_URI_SCHEME, conf.get(OZONE_OM_ADDRESS_KEY));
+ conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
+
+ try (FileSystem fs = FileSystem.get(conf)) {
+ assertTrue(((SafeMode)fs).setSafeMode(SafeModeAction.GET));
+
+ Thread t = new Thread(() -> {
+ try {
+ LOG.info("Sleep 10 seconds and then start DataNodes.");
+ Thread.sleep(10 * 1000);
Review Comment:
Generally we should avoid calling long `sleep()` like this.
Can we use `GenericTestUtils.waitFor()` with much shorter interval to check if the goal is reached before continuing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #6189:
URL: https://github.com/apache/ozone/pull/6189
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1483999123
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java:
##########
@@ -334,4 +343,46 @@ public void testSCMSafeModeDisabled() throws Exception {
cluster.restartStorageContainerManager(true);
assertFalse(scm.isInSafeMode());
}
+
+ @Test
+ public void testCreateRetryWhileSCMSafeMode() throws Exception {
+ // Test1: Test safe mode when there are no containers in system.
+ cluster.stop();
+
+ try {
+ cluster = builder.build();
+ } catch (IOException e) {
+ fail("Cluster startup failed.");
+ }
+
+ final String rootPath = String.format("%s://%s/",
+ OZONE_OFS_URI_SCHEME, conf.get(OZONE_OM_ADDRESS_KEY));
+ conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
+
+ try (FileSystem fs = FileSystem.get(conf)) {
+ assertTrue(((SafeMode)fs).setSafeMode(SafeModeAction.GET));
+
+ Thread t = new Thread(() -> {
+ try {
+ LOG.info("Sleep 10 seconds and then start DataNodes.");
+ Thread.sleep(10 * 1000);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1484133949
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -258,6 +259,10 @@ public final class OzoneManagerProtocolClientSideTranslatorPB
= new ThreadLocal<>();
private boolean s3AuthCheck;
+
+ public static final int BLOCK_ALLOCATION_RETRY_COUNT = 5;
+ public static final int BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS = 3000;
Review Comment:
I'm fine with either way. We can make it configurable later if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1483973646
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java:
##########
@@ -334,4 +343,46 @@ public void testSCMSafeModeDisabled() throws Exception {
cluster.restartStorageContainerManager(true);
assertFalse(scm.isInSafeMode());
}
+
+ @Test
+ public void testCreateRetryWhileSCMSafeMode() throws Exception {
+ // Test1: Test safe mode when there are no containers in system.
+ cluster.stop();
+
+ try {
+ cluster = builder.build();
+ } catch (IOException e) {
+ fail("Cluster startup failed.");
+ }
+
+ final String rootPath = String.format("%s://%s/",
+ OZONE_OFS_URI_SCHEME, conf.get(OZONE_OM_ADDRESS_KEY));
+ conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
+
+ try (FileSystem fs = FileSystem.get(conf)) {
+ assertTrue(((SafeMode)fs).setSafeMode(SafeModeAction.GET));
+
+ Thread t = new Thread(() -> {
+ try {
+ LOG.info("Sleep 10 seconds and then start DataNodes.");
+ Thread.sleep(10 * 1000);
Review Comment:
Generally we should avoid calling long `sleep()` like this.
Can we use `GenericTestUtils.waitFor()` with much shorter interval to check the goal is reached before continuing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1484060705
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -258,6 +259,10 @@ public final class OzoneManagerProtocolClientSideTranslatorPB
= new ThreadLocal<>();
private boolean s3AuthCheck;
+
+ public static final int BLOCK_ALLOCATION_RETRY_COUNT = 5;
+ public static final int BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS = 3000;
Review Comment:
Not exactly sure about the time and it may depend upon the environment load too.
Shall I add new config about retry count and also increase default wait to 30sec ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on PR #6189:
URL: https://github.com/apache/ozone/pull/6189#issuecomment-1935435320
> This code is executed in the OM's request handler when OM receives the request from the client. Client --> OM --> SCM.
>
> IMO It's probably not a good idea to sleep inside a request handler, because if there are many clients, their request will all get blocked because the thread pool for the request handlers is exhausted, which may cause other requests to fail.
>
> Would it make more sense to retry at client side?
Yes makes sense. Moved retry logic at client side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1483084704
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -199,18 +199,29 @@ protected List< OmKeyLocationInfo > allocateBlock(ScmClient scmClient,
List<OmKeyLocationInfo> locationInfos = new ArrayList<>(numBlocks);
String remoteUser = getRemoteUser().getShortUserName();
List<AllocatedBlock> allocatedBlocks;
- try {
- allocatedBlocks = scmClient.getBlockClient()
- .allocateBlock(scmBlockSize, numBlocks, replicationConfig, serviceID,
- excludeList, clientMachine);
- } catch (SCMException ex) {
- omMetrics.incNumBlockAllocateCallFails();
- if (ex.getResult()
- .equals(SCMException.ResultCodes.SAFE_MODE_EXCEPTION)) {
- throw new OMException(ex.getMessage(),
- OMException.ResultCodes.SCM_IN_SAFE_MODE);
+ int retryCount = 5;
+ while (true) {
+ try {
+ allocatedBlocks = scmClient.getBlockClient()
+ .allocateBlock(scmBlockSize, numBlocks, replicationConfig, serviceID,
+ excludeList, clientMachine);
+ } catch (SCMException ex) {
+ omMetrics.incNumBlockAllocateCallFails();
+ if (ex.getResult().equals(SCMException.ResultCodes.SAFE_MODE_EXCEPTION) && retryCount > 0) {
+ retryCount--;
+ // SCM is in safe mode, retry again
Review Comment:
Maybe add a debug log saying " allocate block failed as SCM is in safe mode , x retries remaining"
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -199,18 +199,29 @@ protected List< OmKeyLocationInfo > allocateBlock(ScmClient scmClient,
List<OmKeyLocationInfo> locationInfos = new ArrayList<>(numBlocks);
String remoteUser = getRemoteUser().getShortUserName();
List<AllocatedBlock> allocatedBlocks;
- try {
- allocatedBlocks = scmClient.getBlockClient()
- .allocateBlock(scmBlockSize, numBlocks, replicationConfig, serviceID,
- excludeList, clientMachine);
- } catch (SCMException ex) {
- omMetrics.incNumBlockAllocateCallFails();
- if (ex.getResult()
- .equals(SCMException.ResultCodes.SAFE_MODE_EXCEPTION)) {
- throw new OMException(ex.getMessage(),
- OMException.ResultCodes.SCM_IN_SAFE_MODE);
+ int retryCount = 5;
Review Comment:
Can use a static constant instead
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -199,18 +199,29 @@ protected List< OmKeyLocationInfo > allocateBlock(ScmClient scmClient,
List<OmKeyLocationInfo> locationInfos = new ArrayList<>(numBlocks);
String remoteUser = getRemoteUser().getShortUserName();
List<AllocatedBlock> allocatedBlocks;
- try {
- allocatedBlocks = scmClient.getBlockClient()
- .allocateBlock(scmBlockSize, numBlocks, replicationConfig, serviceID,
- excludeList, clientMachine);
- } catch (SCMException ex) {
- omMetrics.incNumBlockAllocateCallFails();
- if (ex.getResult()
- .equals(SCMException.ResultCodes.SAFE_MODE_EXCEPTION)) {
- throw new OMException(ex.getMessage(),
- OMException.ResultCodes.SCM_IN_SAFE_MODE);
+ int retryCount = 5;
+ while (true) {
+ try {
+ allocatedBlocks = scmClient.getBlockClient()
+ .allocateBlock(scmBlockSize, numBlocks, replicationConfig, serviceID,
+ excludeList, clientMachine);
+ } catch (SCMException ex) {
+ omMetrics.incNumBlockAllocateCallFails();
+ if (ex.getResult().equals(SCMException.ResultCodes.SAFE_MODE_EXCEPTION) && retryCount > 0) {
+ retryCount--;
+ // SCM is in safe mode, retry again
+ try {
+ Thread.sleep(3000);
Review Comment:
can use static constant for 3000 also
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmSafeMode.java:
##########
@@ -334,4 +343,46 @@ public void testSCMSafeModeDisabled() throws Exception {
cluster.restartStorageContainerManager(true);
assertFalse(scm.isInSafeMode());
}
+
+ @Test
+ public void testCreateRetryWhileSCMSafeMode() throws Exception {
+ // Test1: Test safe mode when there are no containers in system.
+ cluster.stop();
+
+ try {
+ cluster = builder.build();
+ } catch (IOException e) {
+ fail("Cluster startup failed.");
+ }
+
+ final String rootPath = String.format("%s://%s/",
+ OZONE_OFS_URI_SCHEME, conf.get(OZONE_OM_ADDRESS_KEY));
+ conf.set(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY, rootPath);
+
+ try (FileSystem fs = FileSystem.get(conf)) {
+ assertTrue(((SafeMode)fs).setSafeMode(SafeModeAction.GET));
+
+ Thread t = new Thread(() -> {
+ try {
+ LOG.info("Sleep 10 seconds and then start DataNodes.");
+ Thread.sleep(10 * 1000);
+
+ cluster.startHddsDatanodes();
+ cluster.waitForClusterToBeReady();
+ cluster.waitTobeOutOfSafeMode();
+ } catch (InterruptedException | TimeoutException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ t.start();
+
+ final Path file = new Path("file");
+ try (FSDataOutputStream outputStream = fs.create(file, true)) {
+ LOG.info("Successfully created a file");
Review Comment:
This can be flaky at times, can we instead verify the omMetric : NumBlockAllocateCallFails is 5 when SCM is in safemode. I'm also okay if the current test passes flaky check if executed multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6189:
URL: https://github.com/apache/ozone/pull/6189#issuecomment-1935586389
Is this change specific to the new feature, i.e. does it need to go into the feature branch instead of `master`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on PR #6189:
URL: https://github.com/apache/ozone/pull/6189#issuecomment-1935696284
> Is this change specific to the new feature, i.e. does it need to go into the feature branch instead of `master`?
It's not related to new feature but issue is seen during hbase testing and impacting hbase.
@jojochuang, Can you please help to confirm whether it can be merged into master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #6189:
URL: https://github.com/apache/ozone/pull/6189#issuecomment-1936448234
IMO it is specific to HBase but I don't think we've experienced this problem with other applications. I'm fine to forward port it to master branch (assuming no code conflicts)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1484019183
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -258,6 +259,10 @@ public final class OzoneManagerProtocolClientSideTranslatorPB
= new ThreadLocal<>();
private boolean s3AuthCheck;
+
+ public static final int BLOCK_ALLOCATION_RETRY_COUNT = 5;
+ public static final int BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS = 3000;
Review Comment:
1. Is 15 seconds enough for SCM to go out of safe mode during rolling restart?
2. Would this be made configurable later?
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -2243,12 +2246,36 @@ public OpenKeySession createFile(OmKeyArgs args,
OMRequest omRequest = createOMRequest(Type.CreateFile)
.setCreateFileRequest(createFileRequest)
.build();
- CreateFileResponse resp =
- handleError(submitRequest(omRequest)).getCreateFileResponse();
+ CreateFileResponse resp = handleSubmitRequestAndSCMSafeModeRetry(omRequest).getCreateFileResponse();
+
return new OpenKeySession(resp.getID(),
OmKeyInfo.getFromProtobuf(resp.getKeyInfo()), resp.getOpenVersion());
}
+
+ @Nonnull
+ private OMResponse handleSubmitRequestAndSCMSafeModeRetry(OMRequest omRequest) throws IOException {
+ int retryCount = BLOCK_ALLOCATION_RETRY_COUNT;
+ while (true) {
+ try {
+ return handleError(submitRequest(omRequest));
+ } catch (OMException e) {
+ if (e.getResult().equals(SCM_IN_SAFE_MODE) && retryCount > 0) {
+ retryCount--;
Review Comment:
Is it possible to print some messages (to stderr) for a CLI client so the user is informed of the wait? Otherwise it would look like the CLI is stuck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1484057577
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -2243,12 +2246,36 @@ public OpenKeySession createFile(OmKeyArgs args,
OMRequest omRequest = createOMRequest(Type.CreateFile)
.setCreateFileRequest(createFileRequest)
.build();
- CreateFileResponse resp =
- handleError(submitRequest(omRequest)).getCreateFileResponse();
+ CreateFileResponse resp = handleSubmitRequestAndSCMSafeModeRetry(omRequest).getCreateFileResponse();
+
return new OpenKeySession(resp.getID(),
OmKeyInfo.getFromProtobuf(resp.getKeyInfo()), resp.getOpenVersion());
}
+
+ @Nonnull
+ private OMResponse handleSubmitRequestAndSCMSafeModeRetry(OMRequest omRequest) throws IOException {
+ int retryCount = BLOCK_ALLOCATION_RETRY_COUNT;
+ while (true) {
+ try {
+ return handleError(submitRequest(omRequest));
+ } catch (OMException e) {
+ if (e.getResult().equals(SCM_IN_SAFE_MODE) && retryCount > 0) {
+ retryCount--;
Review Comment:
Added CLI error message
```
bash-4.2$ ozone sh key put vol1/bucket1/key1 file1
SCM is in safe mode, request will be retried after 3000ms
SCM is in safe mode, request will be retried after 3000ms
SCM is in safe mode, request will be retried after 3000ms
SCM is in safe mode, request will be retried after 3000ms
SCM is in safe mode, request will be retried after 3000ms
SCM_IN_SAFE_MODE SafeModePrecheck failed for allocateBlock
bash-4.2$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1484147074
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -2243,12 +2246,38 @@ public OpenKeySession createFile(OmKeyArgs args,
OMRequest omRequest = createOMRequest(Type.CreateFile)
.setCreateFileRequest(createFileRequest)
.build();
- CreateFileResponse resp =
- handleError(submitRequest(omRequest)).getCreateFileResponse();
+ CreateFileResponse resp = handleSubmitRequestAndSCMSafeModeRetry(omRequest).getCreateFileResponse();
+
return new OpenKeySession(resp.getID(),
OmKeyInfo.getFromProtobuf(resp.getKeyInfo()), resp.getOpenVersion());
}
+
+ @Nonnull
+ private OMResponse handleSubmitRequestAndSCMSafeModeRetry(OMRequest omRequest) throws IOException {
+ int retryCount = BLOCK_ALLOCATION_RETRY_COUNT;
+ while (true) {
+ try {
+ return handleError(submitRequest(omRequest));
+ } catch (OMException e) {
+ if (e.getResult().equals(SCM_IN_SAFE_MODE) && retryCount > 0) {
+ System.err.println("SCM is in safe mode, request will be retried after " +
+ BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS + "ms");
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10256. Retry block allocation when SCM is in safe mode. [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #6189:
URL: https://github.com/apache/ozone/pull/6189#discussion_r1484132704
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java:
##########
@@ -2243,12 +2246,38 @@ public OpenKeySession createFile(OmKeyArgs args,
OMRequest omRequest = createOMRequest(Type.CreateFile)
.setCreateFileRequest(createFileRequest)
.build();
- CreateFileResponse resp =
- handleError(submitRequest(omRequest)).getCreateFileResponse();
+ CreateFileResponse resp = handleSubmitRequestAndSCMSafeModeRetry(omRequest).getCreateFileResponse();
+
return new OpenKeySession(resp.getID(),
OmKeyInfo.getFromProtobuf(resp.getKeyInfo()), resp.getOpenVersion());
}
+
+ @Nonnull
+ private OMResponse handleSubmitRequestAndSCMSafeModeRetry(OMRequest omRequest) throws IOException {
+ int retryCount = BLOCK_ALLOCATION_RETRY_COUNT;
+ while (true) {
+ try {
+ return handleError(submitRequest(omRequest));
+ } catch (OMException e) {
+ if (e.getResult().equals(SCM_IN_SAFE_MODE) && retryCount > 0) {
+ System.err.println("SCM is in safe mode, request will be retried after " +
+ BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS + "ms");
Review Comment:
```suggestion
System.err.println("SCM is in safe mode. Will retry in " +
BLOCK_ALLOCATION_RETRY_WAIT_TIME_MS + "ms");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org