You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/11/10 17:06:52 UTC
[GitHub] [ozone] devmadhuu opened a new pull request, #3947: HDDS-3486. Recon cannot track missing containers that were created an…
devmadhuu opened a new pull request, #3947:
URL: https://github.com/apache/ozone/pull/3947
Root Cause:
When recon is down during the window a new container got created and went missing, recon doesn't have any way to know as data nodes were down and no one sending report to recon, and also recon will not sync with SCM DB due to a condition of if Container number diff between SCM and Recon is greater than threshold value defined in "ozone.recon.scm.container.threshold", then recon SCM rocks DB is updated and sync with scm Rocks DB, however this is only at Recon startup, after that Recon is only dependent on container reports from data nodes. A part from above, there is another design point that SCM rocks DB doesn't get updated with container data if "ozone.scm.ratis.enable" is set as true which is default true and data flushes to SCM rocks DB when ratis snapshot is taken which depends on "dfs.ratis.snapshot.threshold" value (default 10000), so even if we want to do a periodic sync with SCM rocks DB, it will not give any info except SCM Rocks DB is updated or flushed.
Changes done to fix the issue:
Used periodic sync to recon container cache from SCM container cache using SCM listContainer API exposed at rpc port.
https://issues.apache.org/jira/browse/HDDS-3486
## How was this patch tested?
Tested manually using test cases as well as bringing recon down and add a new container in SCM and make it missing container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1330702780
@smengcl @dombizita @kerneltime @sumitagrawl - pls check, all review comments fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1037675706
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -411,22 +460,102 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
Review Comment:
It would be nice to have a unit test for this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created an…
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1310613776
@kerneltime , @smengcl pls review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1047547419
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java:
##########
@@ -89,6 +90,42 @@ public void shutdown() {
}
}
+ @Test
+ public void testSyncSCMContainerInfo() throws Exception {
+ ReconStorageContainerManagerFacade reconScm =
+ (ReconStorageContainerManagerFacade)
+ cluster.getReconServer().getReconStorageContainerManager();
+ StorageContainerManager scm = cluster.getStorageContainerManager();
+ ContainerManager scmContainerManager = scm.getContainerManager();
+ ContainerManager reconContainerManager = reconScm.getContainerManager();
+ final ContainerInfo container1 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ final ContainerInfo container2 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ reconContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ int scmContainersCount = scmContainerManager.getContainers().size();
+ int reconContainersCount = reconContainerManager
+ .getContainers().size();
+ Assert.assertNotEquals(scmContainersCount, reconContainersCount);
+ reconScm.syncWithSCMContainerInfo();
+ scmContainersCount = scmContainerManager.getContainers().size();
Review Comment:
But line 122: " reconScm.syncWithSCMContainerInfo();" only changes the reconContainersCount, not the scmContainersCount.
So line 123 is unnecessary, and confusing, because it makes it seem like syncWithSCMContainerInfo() changes scmContainersCount, but it doesn't.
Isn't that correct? or am I misunderstanding something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
sumitagrawl commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1034508858
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
+ Double.valueOf(containersMetaDataTotalRpcRespSizeMB) /
+ Double.valueOf(DEFAULT_RPC_SIZE);
+ long totalApiCalls = Math.round(Math.ceil(numOfApiCalls));
+ long containerCountPerCall = containersMetaDataTotalRpcRespSizeMB <=
+ DEFAULT_RPC_SIZE ? containerCount : DEFAULT_RPC_SIZE;
+ if (containerCount > 0) {
+ for (int startContainerId = 1; startContainerId <= totalApiCalls;
+ startContainerId++) {
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers(Long.valueOf(startContainerId).longValue(),
+ Long.valueOf(containerCountPerCall).intValue(),
+ HddsProtos.LifeCycleState.CLOSED);
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers from SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ long containerID = containerInfo.getContainerID();
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(
+ containerID);
+ containerWithPipeline.getContainerInfo().setState(
+ HddsProtos.LifeCycleState.UNKNOWN);
Review Comment:
Unknown state reporting is not required, [HDDS-4539](https://issues.apache.org/jira/browse/HDDS-4539) with this when fixed, unknown state will never come.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020083958
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3397,4 +3397,21 @@
optimum performance.
</description>
</property>
+
+ <property>
+ <name>ozone.recon.scm.snapshot.task.initial.delay</name>
+ <value>1m</value>
+ <tag>OZONE, RECON, OM</tag>
Review Comment:
changes done
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3397,4 +3397,21 @@
optimum performance.
</description>
</property>
+
+ <property>
+ <name>ozone.recon.scm.snapshot.task.initial.delay</name>
+ <value>1m</value>
+ <tag>OZONE, RECON, OM</tag>
+ <description>
+ Initial delay in MINUTES by Recon to request SCM DB Snapshot.
+ </description>
+ </property>
+ <property>
+ <name>ozone.recon.scm.snapshot.task.interval.delay</name>
+ <value>10m</value>
Review Comment:
reduced to 24h as frequency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1328788433
@sumitagrawl - pls review once, changes done and pushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1048055541
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java:
##########
@@ -89,6 +90,42 @@ public void shutdown() {
}
}
+ @Test
+ public void testSyncSCMContainerInfo() throws Exception {
+ ReconStorageContainerManagerFacade reconScm =
+ (ReconStorageContainerManagerFacade)
+ cluster.getReconServer().getReconStorageContainerManager();
+ StorageContainerManager scm = cluster.getStorageContainerManager();
+ ContainerManager scmContainerManager = scm.getContainerManager();
+ ContainerManager reconContainerManager = reconScm.getContainerManager();
+ final ContainerInfo container1 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ final ContainerInfo container2 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ reconContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ int scmContainersCount = scmContainerManager.getContainers().size();
+ int reconContainersCount = reconContainerManager
+ .getContainers().size();
+ Assert.assertNotEquals(scmContainersCount, reconContainersCount);
+ reconScm.syncWithSCMContainerInfo();
+ scmContainersCount = scmContainerManager.getContainers().size();
Review Comment:
@GeorgeJahad - Sorry, my bad... I got your point. Its fixed. Pls r-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1026091403
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +449,204 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
Review Comment:
Perhaps this is a better name?
```suggestion
public boolean syncWithSCMContainerInfo()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1032049465
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java:
##########
@@ -91,6 +92,9 @@ public void getMetrics(MetricsCollector collector, boolean all) {
.addGauge(Interns.info("DeletedContainers",
"Number of containers in deleted state"),
stateCount.get(DELETED.toString()))
+ .addGauge(Interns.info("UnknownContainers",
Review Comment:
done
##########
hadoop-ozone/dev-support/intellij/ozone-site.xml:
##########
@@ -71,4 +71,24 @@
<name>datanode.replication.port</name>
<value>0</value>
</property>
+ <property>
+ <name>ozone.scm.ratis.enable</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hdds.container.report.interval</name>
+ <value>60s</value>
+ </property>
+ <property>
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020076177
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
Review Comment:
@swagle @kerneltime , pls provide your comments on above solution suggested by @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1023076453
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +449,204 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
Review Comment:
```suggestion
LOG.debug("Got list of containers from SCM : " +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1023076453
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +449,204 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
Review Comment:
```suggestion
LOG.info("Got list of containers form SCM : " +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1323424068
> Check the container creation logic, if the containers can be ordered in terms of creation time, it might be better to iterate them in a paginated manner. At exabyte of storage under management, the number of containers managed will approach 300 million. It won't be feasible to implement it as a single API to fetch. In essence, what you want to implement is a walk over the container space to identify any missing containers.
Implement using pagination approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1034676940
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
+ Double.valueOf(containersMetaDataTotalRpcRespSizeMB) /
+ Double.valueOf(DEFAULT_RPC_SIZE);
+ long totalApiCalls = Math.round(Math.ceil(numOfApiCalls));
+ long containerCountPerCall = containersMetaDataTotalRpcRespSizeMB <=
+ DEFAULT_RPC_SIZE ? containerCount : DEFAULT_RPC_SIZE;
+ if (containerCount > 0) {
+ for (int startContainerId = 1; startContainerId <= totalApiCalls;
+ startContainerId++) {
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers(Long.valueOf(startContainerId).longValue(),
+ Long.valueOf(containerCountPerCall).intValue(),
+ HddsProtos.LifeCycleState.CLOSED);
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers from SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ long containerID = containerInfo.getContainerID();
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(
+ containerID);
+ containerWithPipeline.getContainerInfo().setState(
+ HddsProtos.LifeCycleState.UNKNOWN);
Review Comment:
@sumitagrawl - pls re-review. Changes pushed as per comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1039382069
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -411,22 +460,102 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
Review Comment:
@GeorgeJahad - Thank you for review. Its added, Pls re-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020075201
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(containerID);
+ containerManager.addNewContainer(containerWithPipeline);
+ } catch (IOException e) {
+ LOG.error("Could not get container with pipeline " +
+ "for container : {}", containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Could not add new container {} in Recon " +
+ "container manager cache.", containerID);
+ }
+ }
+ synchronized (containerInfo) {
+ containerReplicas.forEach(containerReplicaProto -> {
+ final ContainerReplica replica = buildContainerReplica(
+ containerInfo, containerReplicaProto);
+ try {
+ updateContainerState(containerInfo, replica);
+ containerManager.updateContainerReplica(
+ containerInfo.containerID(), replica);
+ } catch (ContainerNotFoundException e) {
+ LOG.error("Could not update container replica as " +
+ "container {} not found.", containerID);
+ } catch (InvalidStateTransitionException e) {
+ LOG.error("Invalid state transition for container {}",
+ containerID);
+ } catch (IOException e) {
+ LOG.error("Could not update container {} state.",
+ containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Timeout while updating container {} state.",
+ containerID);
+ }
+ });
+ }
+ } catch (IOException e) {
+ LOG.error("Unable to get container replicas for container : {}",
+ containerInfo.getContainerID());
+ }
+ });
+ } else {
+ LOG.debug("SCM DB sync is already running.");
+ return false;
+ }
} catch (IOException e) {
LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ return false;
}
- } else {
- LOG.error("Null snapshot location got from SCM.");
}
+ return true;
+ }
+
+ private boolean updateContainerState(ContainerInfo container,
+ ContainerReplica replica)
+ throws InvalidStateTransitionException, IOException, TimeoutException {
+ final ContainerID containerId = container.containerID();
Review Comment:
We need container replica and state as container health task needs that to process for unhealthy containers. @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
sumitagrawl commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1311496737
> @swagle , @kerneltime , pls provide your opinion on solution @sumitagrawl is suggesting
I think we need follow same solution as earlier, new solution have performance issue of getting huge number of container data. Recon will be busy in communication to SCM and SCM will be busy in responding Recon.
For periodic part, We can do flush of transaction at SCM in this special call and then get DB from SCM as old one to avoid load on SCM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1026092119
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +449,204 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
Review Comment:
I think it is ok to log the total containers in info level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1319643914
Check the container creation logic, if the containers can be ordered, it might be better to iterate them in a paginated manner. At exabyte of storage under management, the number of containers managed will approach 300 million. It won't be feasible to implement it as a single API to fetch.
In essence, what you want to implement is a walk over the container space to identify any missing containers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1046296202
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java:
##########
@@ -89,6 +90,42 @@ public void shutdown() {
}
}
+ @Test
+ public void testSyncSCMContainerInfo() throws Exception {
+ ReconStorageContainerManagerFacade reconScm =
+ (ReconStorageContainerManagerFacade)
+ cluster.getReconServer().getReconStorageContainerManager();
+ StorageContainerManager scm = cluster.getStorageContainerManager();
+ ContainerManager scmContainerManager = scm.getContainerManager();
+ ContainerManager reconContainerManager = reconScm.getContainerManager();
+ final ContainerInfo container1 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ final ContainerInfo container2 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ reconContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ int scmContainersCount = scmContainerManager.getContainers().size();
+ int reconContainersCount = reconContainerManager
+ .getContainers().size();
+ Assert.assertNotEquals(scmContainersCount, reconContainersCount);
+ reconScm.syncWithSCMContainerInfo();
+ scmContainersCount = scmContainerManager.getContainers().size();
Review Comment:
Is this line needed? The scmContainerCount shouldn't have changed, should it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
sumitagrawl commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020015205
##########
hadoop-ozone/dev-support/intellij/ozone-site.xml:
##########
@@ -71,4 +71,24 @@
<name>datanode.replication.port</name>
<value>0</value>
</property>
+ <property>
+ <name>ozone.scm.ratis.enable</name>
+ <value>false</value>
+ </property>
Review Comment:
Seems local changes is pushed, plz revert
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3397,4 +3397,21 @@
optimum performance.
</description>
</property>
+
+ <property>
+ <name>ozone.recon.scm.snapshot.task.initial.delay</name>
+ <value>1m</value>
+ <tag>OZONE, RECON, OM</tag>
+ <description>
+ Initial delay in MINUTES by Recon to request SCM DB Snapshot.
+ </description>
+ </property>
+ <property>
+ <name>ozone.recon.scm.snapshot.task.interval.delay</name>
+ <value>10m</value>
Review Comment:
scm query for snapshot can be reduced to once a day after first sync, as Recon can construct same view from DN. This is just a precaution additionally to avoid loa on SCM.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java:
##########
@@ -143,6 +143,19 @@ public final class ReconServerConfigKeys {
"ozone.recon.scm.connection.request.timeout";
public static final String
OZONE_RECON_SCM_CONNECTION_REQUEST_TIMEOUT_DEFAULT = "5s";
+
+ public static final String OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DELAY =
+ "ozone.recon.scm.snapshot.task.interval.delay";
+
+ public static final String OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DEFAULT
+ = "10m";
Review Comment:
same as ozone xml configuration, this periodic sync can be with lesser frequency as once a day
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3397,4 +3397,21 @@
optimum performance.
</description>
</property>
+
+ <property>
+ <name>ozone.recon.scm.snapshot.task.initial.delay</name>
+ <value>1m</value>
+ <tag>OZONE, RECON, OM</tag>
Review Comment:
Tag should be only RECON, right? OM is not applicable
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -315,6 +339,33 @@ public void start() {
} else {
initializePipelinesFromScm();
}
+ LOG.debug("Started the SCM Container Info sync scheduler.");
+ long interval = ozoneConfiguration.getTimeDuration(
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DELAY,
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DEFAULT, TimeUnit.MILLISECONDS);
+ long initialDelay = ozoneConfiguration.getTimeDuration(
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INITIAL_DELAY,
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INITIAL_DELAY_DEFAULT,
+ TimeUnit.MILLISECONDS);
+ // This periodic sync with SCM container cache is needed because during
+ // the window when recon will be down and any container being added
+ // newly and went missing, that container will not be reported as missing by
+ // recon till there is a difference of container count equivalent to
+ // threshold value defined in "ozone.recon.scm.container.threshold"
+ // between SCM container cache and recon container cache.
+ scheduler.scheduleWithFixedDelay(() -> {
Review Comment:
Can we refactor this to background service to perform this as separate class
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
Review Comment:
isSyncDataFromSCMRunning will never be executed as its already set in updateReconSCMDBWithNewSnapshot
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(containerID);
+ containerManager.addNewContainer(containerWithPipeline);
+ } catch (IOException e) {
+ LOG.error("Could not get container with pipeline " +
+ "for container : {}", containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Could not add new container {} in Recon " +
+ "container manager cache.", containerID);
+ }
+ }
+ synchronized (containerInfo) {
Review Comment:
This lock is of no meaning as its on temporary object from data received from SCM
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
Review Comment:
This will make communication stuck for long time, if there are 10,000 of containers or much higher, including closed containers.
We need a kind of FCR report from SCM in a iterative manner to support this with container information.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(containerID);
+ containerManager.addNewContainer(containerWithPipeline);
+ } catch (IOException e) {
+ LOG.error("Could not get container with pipeline " +
+ "for container : {}", containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Could not add new container {} in Recon " +
+ "container manager cache.", containerID);
+ }
+ }
+ synchronized (containerInfo) {
+ containerReplicas.forEach(containerReplicaProto -> {
+ final ContainerReplica replica = buildContainerReplica(
+ containerInfo, containerReplicaProto);
+ try {
+ updateContainerState(containerInfo, replica);
+ containerManager.updateContainerReplica(
+ containerInfo.containerID(), replica);
+ } catch (ContainerNotFoundException e) {
+ LOG.error("Could not update container replica as " +
+ "container {} not found.", containerID);
+ } catch (InvalidStateTransitionException e) {
+ LOG.error("Invalid state transition for container {}",
+ containerID);
+ } catch (IOException e) {
+ LOG.error("Could not update container {} state.",
+ containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Timeout while updating container {} state.",
+ containerID);
+ }
+ });
+ }
+ } catch (IOException e) {
+ LOG.error("Unable to get container replicas for container : {}",
+ containerInfo.getContainerID());
+ }
+ });
+ } else {
+ LOG.debug("SCM DB sync is already running.");
+ return false;
+ }
} catch (IOException e) {
LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ return false;
}
- } else {
- LOG.error("Null snapshot location got from SCM.");
}
+ return true;
+ }
+
+ private boolean updateContainerState(ContainerInfo container,
+ ContainerReplica replica)
+ throws InvalidStateTransitionException, IOException, TimeoutException {
+ final ContainerID containerId = container.containerID();
Review Comment:
I think we need not update container state based on replica, this will be done automatically by FCR report from DN, task here is to get missing container information which is present at SCM, but not present at DN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020086079
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(containerID);
+ containerManager.addNewContainer(containerWithPipeline);
+ } catch (IOException e) {
+ LOG.error("Could not get container with pipeline " +
+ "for container : {}", containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Could not add new container {} in Recon " +
+ "container manager cache.", containerID);
+ }
+ }
+ synchronized (containerInfo) {
Review Comment:
Changes done and fixed @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020084617
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java:
##########
@@ -143,6 +143,19 @@ public final class ReconServerConfigKeys {
"ozone.recon.scm.connection.request.timeout";
public static final String
OZONE_RECON_SCM_CONNECTION_REQUEST_TIMEOUT_DEFAULT = "5s";
+
+ public static final String OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DELAY =
+ "ozone.recon.scm.snapshot.task.interval.delay";
+
+ public static final String OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DEFAULT
+ = "10m";
Review Comment:
Changes done to 24h @sumitagrawl
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -315,6 +339,33 @@ public void start() {
} else {
initializePipelinesFromScm();
}
+ LOG.debug("Started the SCM Container Info sync scheduler.");
+ long interval = ozoneConfiguration.getTimeDuration(
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DELAY,
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INTERVAL_DEFAULT, TimeUnit.MILLISECONDS);
+ long initialDelay = ozoneConfiguration.getTimeDuration(
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INITIAL_DELAY,
+ OZONE_RECON_SCM_SNAPSHOT_TASK_INITIAL_DELAY_DEFAULT,
+ TimeUnit.MILLISECONDS);
+ // This periodic sync with SCM container cache is needed because during
+ // the window when recon will be down and any container being added
+ // newly and went missing, that container will not be reported as missing by
+ // recon till there is a difference of container count equivalent to
+ // threshold value defined in "ozone.recon.scm.container.threshold"
+ // between SCM container cache and recon container cache.
+ scheduler.scheduleWithFixedDelay(() -> {
Review Comment:
Is this really needed ? @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
sumitagrawl commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1031598787
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java:
##########
@@ -91,6 +92,9 @@ public void getMetrics(MetricsCollector collector, boolean all) {
.addGauge(Interns.info("DeletedContainers",
"Number of containers in deleted state"),
stateCount.get(DELETED.toString()))
+ .addGauge(Interns.info("UnknownContainers",
Review Comment:
can avoid metrics for unknown state, as there is no possiblity of such state.
##########
hadoop-ozone/dev-support/intellij/ozone-site.xml:
##########
@@ -71,4 +71,24 @@
<name>datanode.replication.port</name>
<value>0</value>
</property>
+ <property>
+ <name>ozone.scm.ratis.enable</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hdds.container.report.interval</name>
+ <value>60s</value>
+ </property>
+ <property>
Review Comment:
we can have the config with default value and respective dev-config can be changed as required while test.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
Review Comment:
Can get actual hadoop RPC size from conf with default value
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
+ Double.valueOf(containersMetaDataTotalRpcRespSizeMB) /
+ Double.valueOf(DEFAULT_RPC_SIZE);
+ long totalApiCalls = Math.round(Math.ceil(numOfApiCalls));
+ long containerCountPerCall = containersMetaDataTotalRpcRespSizeMB <=
+ DEFAULT_RPC_SIZE ? containerCount : DEFAULT_RPC_SIZE;
+ if (containerCount > 0) {
+ for (int startContainerId = 1; startContainerId <= totalApiCalls;
+ startContainerId++) {
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers(Long.valueOf(startContainerId).longValue(),
+ Long.valueOf(containerCountPerCall).intValue(),
+ HddsProtos.LifeCycleState.CLOSED);
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers from SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ long containerID = containerInfo.getContainerID();
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(
+ containerID);
+ containerWithPipeline.getContainerInfo().setState(
+ HddsProtos.LifeCycleState.UNKNOWN);
Review Comment:
Unknown state of container can be removed, its health report to differentiate if Unknown or missing if required.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
+ Double.valueOf(containersMetaDataTotalRpcRespSizeMB) /
+ Double.valueOf(DEFAULT_RPC_SIZE);
+ long totalApiCalls = Math.round(Math.ceil(numOfApiCalls));
+ long containerCountPerCall = containersMetaDataTotalRpcRespSizeMB <=
+ DEFAULT_RPC_SIZE ? containerCount : DEFAULT_RPC_SIZE;
+ if (containerCount > 0) {
+ for (int startContainerId = 1; startContainerId <= totalApiCalls;
Review Comment:
startContainerId should start with 0 index excluding, to get next set of list, and next startIndex should be last queried containerId, this need fix at SCM and recon.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
+ Double.valueOf(containersMetaDataTotalRpcRespSizeMB) /
+ Double.valueOf(DEFAULT_RPC_SIZE);
+ long totalApiCalls = Math.round(Math.ceil(numOfApiCalls));
+ long containerCountPerCall = containersMetaDataTotalRpcRespSizeMB <=
+ DEFAULT_RPC_SIZE ? containerCount : DEFAULT_RPC_SIZE;
+ if (containerCount > 0) {
+ for (int startContainerId = 1; startContainerId <= totalApiCalls;
+ startContainerId++) {
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers(Long.valueOf(startContainerId).longValue(),
+ Long.valueOf(containerCountPerCall).intValue(),
+ HddsProtos.LifeCycleState.CLOSED);
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers from SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ long containerID = containerInfo.getContainerID();
+ boolean isContainerPresentAtRecon =
Review Comment:
Avoid removal of container, as replica information will be lost as registered by DN in periodic call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1031652322
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ try {
+ long containerID = containerInfo.getContainerID();
+ List<HddsProtos.SCMContainerReplicaProto> containerReplicas
+ = scmServiceProvider.getContainerReplicas(containerID);
+ boolean isContainerPresentAtRecon =
+ containers.remove(containerID);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(containerID);
+ containerManager.addNewContainer(containerWithPipeline);
+ } catch (IOException e) {
+ LOG.error("Could not get container with pipeline " +
+ "for container : {}", containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Could not add new container {} in Recon " +
+ "container manager cache.", containerID);
+ }
+ }
+ synchronized (containerInfo) {
+ containerReplicas.forEach(containerReplicaProto -> {
+ final ContainerReplica replica = buildContainerReplica(
+ containerInfo, containerReplicaProto);
+ try {
+ updateContainerState(containerInfo, replica);
+ containerManager.updateContainerReplica(
+ containerInfo.containerID(), replica);
+ } catch (ContainerNotFoundException e) {
+ LOG.error("Could not update container replica as " +
+ "container {} not found.", containerID);
+ } catch (InvalidStateTransitionException e) {
+ LOG.error("Invalid state transition for container {}",
+ containerID);
+ } catch (IOException e) {
+ LOG.error("Could not update container {} state.",
+ containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Timeout while updating container {} state.",
+ containerID);
+ }
+ });
+ }
+ } catch (IOException e) {
+ LOG.error("Unable to get container replicas for container : {}",
+ containerInfo.getContainerID());
+ }
+ });
+ } else {
+ LOG.debug("SCM DB sync is already running.");
+ return false;
+ }
} catch (IOException e) {
LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ return false;
}
- } else {
- LOG.error("Null snapshot location got from SCM.");
}
+ return true;
+ }
+
+ private boolean updateContainerState(ContainerInfo container,
+ ContainerReplica replica)
+ throws InvalidStateTransitionException, IOException, TimeoutException {
+ final ContainerID containerId = container.containerID();
Review Comment:
Removed call to container replica from SCM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1032049561
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +445,90 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long containerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ // Assumption of size of 1 container info object here is 1 MB
+ long containersMetaDataTotalRpcRespSizeMB =
+ CONTAINER_METADATA_SIZE * containerCount;
+ double numOfApiCalls =
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1029134234
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +449,204 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
Review Comment:
fixed
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -396,22 +449,204 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers();
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers frm SCM : " +
Review Comment:
taken care
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1323425116
@sumitagrawl @nandakumar131 @kerneltime , @smengcl - pls re-review. Comments being fixed and tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020084870
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -400,18 +451,200 @@ private void initializeSCMDB() {
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncSCMContainerInfoWithReconContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
Review Comment:
Changes done and fixed @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020083998
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3397,4 +3397,21 @@
optimum performance.
</description>
</property>
+
+ <property>
+ <name>ozone.recon.scm.snapshot.task.initial.delay</name>
+ <value>1m</value>
+ <tag>OZONE, RECON, OM</tag>
+ <description>
+ Initial delay in MINUTES by Recon to request SCM DB Snapshot.
+ </description>
+ </property>
+ <property>
+ <name>ozone.recon.scm.snapshot.task.interval.delay</name>
+ <value>10m</value>
Review Comment:
reduced to 24h as frequency @sumitagrawl
##########
hadoop-hdds/common/src/main/resources/ozone-default.xml:
##########
@@ -3397,4 +3397,21 @@
optimum performance.
</description>
</property>
+
+ <property>
+ <name>ozone.recon.scm.snapshot.task.initial.delay</name>
+ <value>1m</value>
+ <tag>OZONE, RECON, OM</tag>
Review Comment:
changes done @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1020054082
##########
hadoop-ozone/dev-support/intellij/ozone-site.xml:
##########
@@ -71,4 +71,24 @@
<name>datanode.replication.port</name>
<value>0</value>
</property>
+ <property>
+ <name>ozone.scm.ratis.enable</name>
+ <value>false</value>
+ </property>
Review Comment:
No this is for dev support , so these changes will be helpful. Not done by mistake @sumitagrawl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1051913352
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java:
##########
@@ -411,22 +460,102 @@ private void initializeSCMDB() {
}
} catch (IOException e) {
LOG.error("Exception encountered while getting SCM DB.");
+ } finally {
+ isSyncDataFromSCMRunning.compareAndSet(true, false);
}
}
public void updateReconSCMDBWithNewSnapshot() throws IOException {
- DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
- if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
- LOG.info("Got new checkpoint from SCM : " +
- dbSnapshot.getCheckpointLocation());
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
+ DBCheckpoint dbSnapshot = scmServiceProvider.getSCMDBSnapshot();
+ if (dbSnapshot != null && dbSnapshot.getCheckpointLocation() != null) {
+ LOG.info("Got new checkpoint from SCM : " +
+ dbSnapshot.getCheckpointLocation());
+ try {
+ initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ } catch (IOException e) {
+ LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ }
+ } else {
+ LOG.error("Null snapshot location got from SCM.");
+ }
+ } else {
+ LOG.warn("SCM DB sync is already running.");
+ }
+ }
+
+ public boolean syncWithSCMContainerInfo()
+ throws IOException {
+ if (isSyncDataFromSCMRunning.compareAndSet(false, true)) {
try {
- initializeNewRdbStore(dbSnapshot.getCheckpointLocation().toFile());
+ List<ContainerInfo> containers = containerManager.getContainers();
+
+ long totalContainerCount = scmServiceProvider.getContainerCount(
+ HddsProtos.LifeCycleState.CLOSED);
+ long containerCountPerCall =
+ getContainerCountPerCall(totalContainerCount);
+ long startContainerId = 1;
+ long retrievedContainerCount = 0;
+ if (totalContainerCount > 0) {
+ while (retrievedContainerCount < totalContainerCount) {
+ List<ContainerInfo> listOfContainers = scmServiceProvider.
+ getListOfContainers(startContainerId,
+ Long.valueOf(containerCountPerCall).intValue(),
+ HddsProtos.LifeCycleState.CLOSED);
+ if (null != listOfContainers && listOfContainers.size() > 0) {
+ LOG.info("Got list of containers from SCM : " +
+ listOfContainers.size());
+ listOfContainers.forEach(containerInfo -> {
+ long containerID = containerInfo.getContainerID();
+ boolean isContainerPresentAtRecon =
+ containers.contains(containerInfo);
+ if (!isContainerPresentAtRecon) {
+ try {
+ ContainerWithPipeline containerWithPipeline =
+ scmServiceProvider.getContainerWithPipeline(
+ containerID);
+ containerManager.addNewContainer(containerWithPipeline);
+ } catch (IOException e) {
+ LOG.error("Could not get container with pipeline " +
+ "for container : {}", containerID);
+ } catch (TimeoutException e) {
+ LOG.error("Could not add new container {} in Recon " +
+ "container manager cache.", containerID);
+ }
+ }
+ });
+ startContainerId = listOfContainers.get(
+ listOfContainers.size() - 1).getContainerID() + 1;
+ } else {
+ LOG.info("No containers found at SCM in CLOSED state");
+ return false;
+ }
+ retrievedContainerCount += containerCountPerCall;
+ }
+ }
} catch (IOException e) {
LOG.error("Unable to refresh Recon SCM DB Snapshot. ", e);
+ return false;
}
} else {
- LOG.error("Null snapshot location got from SCM.");
+ LOG.debug("SCM DB sync is already running.");
+ return false;
}
+ return true;
+ }
+
+ private long getContainerCountPerCall(long totalContainerCount) {
+ // Assumption of size of 1 container info object here is 1 MB
Review Comment:
That seems pretty high. SCM footprint per container is in kilobytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1048076393
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java:
##########
@@ -89,6 +90,42 @@ public void shutdown() {
}
}
+ @Test
+ public void testSyncSCMContainerInfo() throws Exception {
+ ReconStorageContainerManagerFacade reconScm =
+ (ReconStorageContainerManagerFacade)
+ cluster.getReconServer().getReconStorageContainerManager();
+ StorageContainerManager scm = cluster.getStorageContainerManager();
+ ContainerManager scmContainerManager = scm.getContainerManager();
+ ContainerManager reconContainerManager = reconScm.getContainerManager();
+ final ContainerInfo container1 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ final ContainerInfo container2 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ reconContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ int scmContainersCount = scmContainerManager.getContainers().size();
+ int reconContainersCount = reconContainerManager
+ .getContainers().size();
+ Assert.assertNotEquals(scmContainersCount, reconContainersCount);
+ reconScm.syncWithSCMContainerInfo();
+ scmContainersCount = scmContainerManager.getContainers().size();
Review Comment:
The test LGTM. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1047055234
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java:
##########
@@ -89,6 +90,42 @@ public void shutdown() {
}
}
+ @Test
+ public void testSyncSCMContainerInfo() throws Exception {
+ ReconStorageContainerManagerFacade reconScm =
+ (ReconStorageContainerManagerFacade)
+ cluster.getReconServer().getReconStorageContainerManager();
+ StorageContainerManager scm = cluster.getStorageContainerManager();
+ ContainerManager scmContainerManager = scm.getContainerManager();
+ ContainerManager reconContainerManager = reconScm.getContainerManager();
+ final ContainerInfo container1 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ final ContainerInfo container2 = scmContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ reconContainerManager.allocateContainer(
+ RatisReplicationConfig.getInstance(
+ HddsProtos.ReplicationFactor.ONE), "admin");
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.FINALIZE);
+ scmContainerManager.updateContainerState(container1.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ scmContainerManager.updateContainerState(container2.containerID(),
+ HddsProtos.LifeCycleEvent.CLOSE);
+ int scmContainersCount = scmContainerManager.getContainers().size();
+ int reconContainersCount = reconContainerManager
+ .getContainers().size();
+ Assert.assertNotEquals(scmContainersCount, reconContainersCount);
+ reconScm.syncWithSCMContainerInfo();
+ scmContainersCount = scmContainerManager.getContainers().size();
Review Comment:
@GeorgeJahad , yes this is the part of test. Check from line 118 to 126, at line 118, scmContainerCount taken and at line 119 reconContainersCount taken and AssertNotEquals should pass as they should not be equal in this specific test. After that at line 122, sync call happens with SCM for containers and then again same assert being repeated for equals, so after sync, assertEquals test will pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1311492541
@swagle , @kerneltime , pls provide your opinion on solution @sumitagrawl is suggesting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on PR #3947:
URL: https://github.com/apache/ozone/pull/3947#issuecomment-1311511494
> > @swagle , @kerneltime , pls provide your opinion on solution @sumitagrawl is suggesting
>
> I think we need follow same solution as earlier, new solution have performance issue of getting huge number of container data. Recon will be busy in communication to SCM and SCM will be busy in responding Recon.
>
> For periodic part, We can do flush of transaction at SCM in this special call and then get DB from SCM as old one to avoid load on SCM.
@swagle - can we reduce sync frequency to 24h and go with current solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 merged pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
nandakumar131 merged PR #3947:
URL: https://github.com/apache/ozone/pull/3947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] devmadhuu commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
devmadhuu commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1033681309
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMContainerMetrics.java:
##########
@@ -78,7 +78,9 @@ public void testSCMContainerMetrics() {
"Number of containers in deleting state"), 6);
verify(mb, times(1)).addGauge(Interns.info("DeletedContainers",
"Number of containers in deleted state"), 7);
+ verify(mb, times(1)).addGauge(Interns.info("UnknownContainers",
Review Comment:
@dombizita - This will not fail, as I have take care not to add Unknown state as part of metrics, so no need to add above line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] dombizita commented on a diff in pull request #3947: HDDS-3486. Recon cannot track missing containers that were created and went missing while it is down.
Posted by GitBox <gi...@apache.org>.
dombizita commented on code in PR #3947:
URL: https://github.com/apache/ozone/pull/3947#discussion_r1033566311
##########
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMContainerMetrics.java:
##########
@@ -78,7 +78,9 @@ public void testSCMContainerMetrics() {
"Number of containers in deleting state"), 6);
verify(mb, times(1)).addGauge(Interns.info("DeletedContainers",
"Number of containers in deleted state"), 7);
+ verify(mb, times(1)).addGauge(Interns.info("UnknownContainers",
Review Comment:
Could this cause the **basic (unit)** [failure](https://github.com/apache/ozone/actions/runs/3565038544/jobs/5989724741)? I think the `"UnknownContainers"` should be added [here](https://github.com/apache/ozone/blob/master/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMContainerMetrics.java#L75L96).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org