You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/03/23 22:21:19 UTC

[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

vivekratnavel opened a new pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714
 
 
   ## What changes were proposed in this pull request?
   
   - Add an endpoint (/api/v1/containers/missing) to serve missing containers response
   - Add unit tests
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-3164
   
   ## How was this patch tested?
   
   - Unit test
   - Manual test: 
     - Bring ozone via docker compose
     - Add random keys with RATIS factor ONE using freon (ozone freon rk --replicationType=RATIS --numOfVolumes=10 --numOfBuckets=10 --numOfKeys=10 --factor=ONE --numOfThreads=20)
     - Bring down one datanode
     - After 10m, the datanode should be reported as dead in Recon and the next run of MissingContainerTask should catch the missing containers in the dead datanode. 
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#discussion_r397504203
 
 

 ##########
 File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/fsck/MissingContainerTask.java
 ##########
 @@ -47,7 +48,7 @@
 
   private ContainerManager containerManager;
   private MissingContainersDao missingContainersDao;
-  private static final long INTERVAL = 5 * 60 * 1000L;
 
 Review comment:
   Good catch :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] vivekratnavel commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#issuecomment-604043200
 
 
   Acceptance test failure is not related to the change. It fails due to no space available on disk
   ```
   Failing this attempt.Diagnostics: [2020-03-25 11:34:00.461]No space available in any of the local directories.
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx merged pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
avijayanhwx merged pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#issuecomment-604104783
 
 
   Thank you for the contribution @vivekratnavel. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#discussion_r397476365
 
 

 ##########
 File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestContainerEndpoint.java
 ##########
 @@ -350,24 +407,52 @@ public void testGetContainersWithPrevKey() {
     assertEquals(1L, containerMetadata.getContainerID());
 
     // test for negative cases
-    response = containerKeyService.getContainers(-1, 5L);
+    response = containerEndpoint.getContainers(-1, 5L);
     responseObject = (ContainersResponse) response.getEntity();
     data = responseObject.getContainersResponseData();
     containers = new ArrayList<>(data.getContainers());
     assertEquals(0, containers.size());
     assertEquals(2, data.getTotalCount());
 
-    response = containerKeyService.getContainers(-1, -1L);
+    response = containerEndpoint.getContainers(-1, -1L);
     responseObject = (ContainersResponse) response.getEntity();
     data = responseObject.getContainersResponseData();
     containers = new ArrayList<>(data.getContainers());
     assertEquals(2, containers.size());
     assertEquals(2, data.getTotalCount());
   }
 
+  @Test
+  public void testGetMissingContainers() {
+    Response response = containerEndpoint.getMissingContainers();
+
+    MissingContainersResponse responseObject =
+        (MissingContainersResponse) response.getEntity();
+
+    assertEquals(0, responseObject.getTotalCount());
+    assertEquals(Collections.EMPTY_LIST, responseObject.getContainers());
+
+    // Add missing containers to the database
+    long missingSince = System.currentTimeMillis();
+    MissingContainers newRecord =
+        new MissingContainers(1L, missingSince);
+    missingContainersDao.insert(newRecord);
+
+    response = containerEndpoint.getMissingContainers();
+    responseObject = (MissingContainersResponse) response.getEntity();
+    assertEquals(1, responseObject.getTotalCount());
+    MissingContainerMetadata container =
+        responseObject.getContainers().stream().findFirst().orElse(null);
+    Assert.assertNotNull(container);
+
+    assertEquals(containerID.getId(), container.getContainerID());
+    assertEquals(keyCount, container.getKeys());
+    assertEquals(pipelineID.getId(), container.getPipelineID());
+    assertEquals(0, container.getDatanodes().size());
+    assertTrue(System.currentTimeMillis() > container.getMissingSince());
 
 Review comment:
   Why not do assertEquals check with "missingSince" ? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#discussion_r397471038
 
 

 ##########
 File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/fsck/TestMissingContainerTask.java
 ##########
 @@ -90,15 +102,19 @@ public void testRun() throws Exception {
     missingContainerTask.start();
 
     LambdaTestUtils.await(6000, 1000, () ->
-        (missingContainersTableHandle.findAll().size() == 1));
+        (missingContainersTableHandle.findAll().size() == 2));
     all = missingContainersTableHandle.findAll();
-    Assert.assertEquals(3, all.get(0).getContainerId().longValue());
-
+    // Container IDs 2 and 3 should be present in the missing containers table
+    Set<Long> missingContainerIDs = Collections.unmodifiableSet(
+        new HashSet<>(Arrays.asList(2L, 3L))
+    );
+    Assert.assertTrue(all.stream().allMatch(record ->
+        missingContainerIDs.stream().anyMatch(
 
 Review comment:
   Nit. This may be replaced with something like 
   
   `all.forEach(r -> Assert.assertTrue(missingContainerIDs.contains(r.getContainerId())));`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] vivekratnavel commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#issuecomment-602888078
 
 
   @avijayanhwx @elek Please review

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#discussion_r397477856
 
 

 ##########
 File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestContainerEndpoint.java
 ##########
 @@ -45,66 +48,120 @@
 import org.apache.hadoop.ozone.recon.api.types.ContainersResponse;
 import org.apache.hadoop.ozone.recon.api.types.KeyMetadata;
 import org.apache.hadoop.ozone.recon.api.types.KeysResponse;
+import org.apache.hadoop.ozone.recon.api.types.MissingContainerMetadata;
+import org.apache.hadoop.ozone.recon.api.types.MissingContainersResponse;
 import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.scm.ReconContainerManager;
+import org.apache.hadoop.ozone.recon.scm.ReconStorageContainerManagerFacade;
 import org.apache.hadoop.ozone.recon.spi.ContainerDBServiceProvider;
+import org.apache.hadoop.ozone.recon.spi.StorageContainerServiceProvider;
 import org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl;
+import org.apache.hadoop.ozone.recon.spi.impl.StorageContainerServiceProviderImpl;
 import org.apache.hadoop.ozone.recon.tasks.ContainerKeyMapperTask;
 import org.apache.hadoop.hdds.utils.db.Table;
+import org.hadoop.ozone.recon.schema.ReconTaskSchemaDefinition;
 import org.hadoop.ozone.recon.schema.StatsSchemaDefinition;
-import org.jooq.impl.DSL;
-import org.jooq.impl.DefaultConfiguration;
+import org.hadoop.ozone.recon.schema.UtilizationSchemaDefinition;
+import org.hadoop.ozone.recon.schema.tables.daos.MissingContainersDao;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.MissingContainers;
+import org.jooq.Configuration;
+import org.junit.Assert;
 import org.junit.Before;
 import org.junit.Test;
 
 import com.google.inject.AbstractModule;
 import com.google.inject.Injector;
 
 /**
- * Test for container key service.
+ * Test for container endpoint.
  */
-public class TestContainerKeyService extends AbstractOMMetadataManagerTest {
+public class TestContainerEndpoint extends AbstractOMMetadataManagerTest {
 
   private ContainerDBServiceProvider containerDbServiceProvider;
-  private Injector injector;
-  private OzoneManagerServiceProviderImpl ozoneManagerServiceProvider;
-  private ContainerKeyService containerKeyService;
+  private ContainerEndpoint containerEndpoint;
   private GuiceInjectorUtilsForTestsImpl guiceInjectorTest =
       new GuiceInjectorUtilsForTestsImpl();
   private boolean isSetupDone = false;
   private ReconOMMetadataManager reconOMMetadataManager;
+  private MissingContainersDao missingContainersDao;
+  private ContainerID containerID = new ContainerID(1L);
+  private PipelineID pipelineID;
+  private long keyCount = 5L;
   private void initializeInjector() throws Exception {
     reconOMMetadataManager = getTestMetadataManager(
         initializeNewOmMetadataManager());
-    ozoneManagerServiceProvider = getMockOzoneManagerServiceProvider();
+    OzoneManagerServiceProviderImpl ozoneManagerServiceProvider =
 
 Review comment:
   Nit. We can probably get rid of this method 'getMockOzoneManagerServiceProvider'.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#discussion_r397467761
 
 

 ##########
 File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/fsck/MissingContainerTask.java
 ##########
 @@ -47,7 +48,7 @@
 
   private ContainerManager containerManager;
   private MissingContainersDao missingContainersDao;
-  private static final long INTERVAL = 5 * 60 * 1000L;
 
 Review comment:
   Unintended change?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.

Posted by GitBox <gi...@apache.org>.
elek commented on issue #714: HDDS-3164. Add Recon endpoint to serve missing containers and its metadata.
URL: https://github.com/apache/hadoop-ozone/pull/714#issuecomment-604987204
 
 
   If you see a flaky test, please disable it (without and issue) + create a new open issue and repeat the build. There is some risk that one flakiness hides a other. Merging patches without green build makes harder to debug flaky tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org