You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2020/08/03 18:45:09 UTC

[GitHub] [accumulo] cradal opened a new pull request #1670: WIP #1631 - Tablet in memory compaction ID is updated before metadata write

cradal opened a new pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670


   @keith-turner  What about something like this? The tablet compaction ID should only be set after a successful Metadata table update.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] cradal commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
cradal commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-706295962


   @keith-turner  and @ctubbsii. I moved the call to tablet.setCompactionID from out of the synchronized block and put it after the id is updated in the metadata table. The compactionId is passed into method that contains the sync'd block. I did not foresee any issues with setting the tablet compactionId after the lock is released.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-707832801


   > @ctubbsii The QA build failed while trying to set up JDK 11? Never seen this before.
   
   Weird. I restarted it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
keith-turner commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-707182180


   > The compactionId is passed into method that contains the sync'd block. I did not foresee any issues with setting the tablet compactionId after the lock is released.
   
   Yeah I think that is ok too.  The setLastCompactionID() method is synchronizing on the same lock as the sync block it was moved from.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] cradal commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
cradal commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r493615509



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableUtils.java
##########
@@ -441,10 +441,17 @@ private UserCompactionHelper(CompactionConfig compactionConfig, Tablet tablet,
       }
 
       if (selectedFiles.isEmpty()) {
-        tablet.setLastCompactionID(compactionId);
-
-        MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
-            tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        Boolean updateMetadataTabletFail = false;
+        try {
+          MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
+              tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        } catch (Exception exc) {
+          updateMetadataTabletFail = true;
+          log.error(exc.getMessage());
+        }

Review comment:
       I see what you mean. updateTabletCompactID does not throw any exception. I thought that this would guarantee no attempts to set the LastCompactionID if the Metadata table update failed. Thinking it over, simply reordering the method calls should work.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] cradal commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
cradal commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-696794340


   @ctubbsii Just a reminder that this PR is still available for review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
keith-turner commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-705739580


   @cradal  and @ctubbsii  the current changes look good.  To close out #1631  I think one more change is needed.  At the following line there is also a call to update the compaction Id in memory.  
   
   https://github.com/apache/accumulo/blob/80ee9ca8092dc4bf07aa1a046d6e668f0e16300e/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L449
   
   A few lines later  `MasterMetadataUtil.replaceDatafiles()` is called which updates the compaction id in the metadata table.  Should probably update memory after the metadata update.
   
   https://github.com/apache/accumulo/blob/80ee9ca8092dc4bf07aa1a046d6e668f0e16300e/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L457


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r492874398



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableUtils.java
##########
@@ -441,10 +441,17 @@ private UserCompactionHelper(CompactionConfig compactionConfig, Tablet tablet,
       }
 
       if (selectedFiles.isEmpty()) {
-        tablet.setLastCompactionID(compactionId);
-
-        MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
-            tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        Boolean updateMetadataTabletFail = false;

Review comment:
       Should use primitive, rather than boxed form.
   
   ```suggestion
           boolean updateMetadataTabletFail = false;
   ```

##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableUtils.java
##########
@@ -441,10 +441,17 @@ private UserCompactionHelper(CompactionConfig compactionConfig, Tablet tablet,
       }
 
       if (selectedFiles.isEmpty()) {
-        tablet.setLastCompactionID(compactionId);
-
-        MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
-            tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        Boolean updateMetadataTabletFail = false;
+        try {
+          MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
+              tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        } catch (Exception exc) {
+          updateMetadataTabletFail = true;
+          log.error(exc.getMessage());
+        }

Review comment:
       This catches exceptions that were previously thrown. Is this a valid change in behavior? Why wouldn't it be sufficient to simply reorder these lines, and let the exception be thrown?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] cradal commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
cradal commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-696794340


   @ctubbsii Just a reminder that this PR is still available for review.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii merged pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii merged pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r496862173



##########
File path: core/src/main/java/org/apache/accumulo/core/metadata/schema/TabletsMetadata.java
##########
@@ -371,6 +371,10 @@ private TabletsMetadata(Scanner scanner, Iterable<TabletMetadata> tmi) {
     this.tablets = tmi;
   }
 
+  public TabletsMetadata() {
+
+  }
+

Review comment:
       Is this needed?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r503361350



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
##########
@@ -458,6 +457,7 @@ StoredTabletFile bringMajorCompactionOnline(Set<StoredTabletFile> oldDatafiles,
         filesInUseByScans, newFile, compactionId, dfv,
         tablet.getTabletServer().getClientAddressString(), lastLocation,
         tablet.getTabletServer().getLock());
+    tablet.setLastCompactionID(compactionId);

Review comment:
       Actually that comment is still relevant.  The tablet's list of files are still being updated before the the metadata table update is made.  For minor compactions the tablets list of files is updated after the metadata update. The comment was referring to when the in memory list of files is updated.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r502615753



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
##########
@@ -458,6 +457,7 @@ StoredTabletFile bringMajorCompactionOnline(Set<StoredTabletFile> oldDatafiles,
         filesInUseByScans, newFile, compactionId, dfv,
         tablet.getTabletServer().getClientAddressString(), lastLocation,
         tablet.getTabletServer().getLock());
+    tablet.setLastCompactionID(compactionId);

Review comment:
       I believe there is a related comment a few lines above that can probably be removed, about `known consistency issue between minor and major compactions`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-700814713


   @keith-turner If you have a chance, would you mind taking a look at this, to see if it's reasonable?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r492874398



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableUtils.java
##########
@@ -441,10 +441,17 @@ private UserCompactionHelper(CompactionConfig compactionConfig, Tablet tablet,
       }
 
       if (selectedFiles.isEmpty()) {
-        tablet.setLastCompactionID(compactionId);
-
-        MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
-            tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        Boolean updateMetadataTabletFail = false;

Review comment:
       Should use primitive, rather than boxed form.
   
   ```suggestion
           boolean updateMetadataTabletFail = false;
   ```

##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableUtils.java
##########
@@ -441,10 +441,17 @@ private UserCompactionHelper(CompactionConfig compactionConfig, Tablet tablet,
       }
 
       if (selectedFiles.isEmpty()) {
-        tablet.setLastCompactionID(compactionId);
-
-        MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
-            tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        Boolean updateMetadataTabletFail = false;
+        try {
+          MetadataTableUtil.updateTabletCompactID(tablet.getExtent(), compactionId,
+              tablet.getTabletServer().getContext(), tablet.getTabletServer().getLock());
+        } catch (Exception exc) {
+          updateMetadataTabletFail = true;
+          log.error(exc.getMessage());
+        }

Review comment:
       This catches exceptions that were previously thrown. Is this a valid change in behavior? Why wouldn't it be sufficient to simply reorder these lines, and let the exception be thrown?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] cradal commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
cradal commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-707802022


   @ctubbsii The QA build failed while trying to set up JDK 11? Never seen this before.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
keith-turner commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-705739580


   @cradal  and @ctubbsii  the current changes look good.  To close out #1631  I think one more change is needed.  At the following line there is also a call to update the compaction Id in memory.  
   
   https://github.com/apache/accumulo/blob/80ee9ca8092dc4bf07aa1a046d6e668f0e16300e/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L449
   
   A few lines later  `MasterMetadataUtil.replaceDatafiles()` is called which updates the compaction id in the metadata table.  Should probably update memory after the metadata update.
   
   https://github.com/apache/accumulo/blob/80ee9ca8092dc4bf07aa1a046d6e668f0e16300e/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java#L457


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#issuecomment-707873538


   Thanks, @cradal !


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #1670: #1631 - Tablet in memory compaction ID is updated before metadata write

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #1670:
URL: https://github.com/apache/accumulo/pull/1670#discussion_r503617670



##########
File path: server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/DatafileManager.java
##########
@@ -458,6 +457,7 @@ StoredTabletFile bringMajorCompactionOnline(Set<StoredTabletFile> oldDatafiles,
         filesInUseByScans, newFile, compactionId, dfv,
         tablet.getTabletServer().getClientAddressString(), lastLocation,
         tablet.getTabletServer().getLock());
+    tablet.setLastCompactionID(compactionId);

Review comment:
       Oh my mistake.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org