You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/11/18 22:08:18 UTC

[GitHub] [hadoop] xkrogen commented on a diff in pull request #4744: HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer

xkrogen commented on code in PR #4744:
URL: https://github.com/apache/hadoop/pull/4744#discussion_r1026900094


##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java:
##########
@@ -283,7 +283,7 @@ public void stop() throws IOException {
   }
   
   @VisibleForTesting
-  FSEditLog getEditLog() {
+  public FSEditLog getEditLog() {

Review Comment:
   why `public`? can we leave it package-private? it seems like it's not used anywhere AFAICT



##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestNNWithQJM.java:
##########
@@ -197,10 +197,9 @@ public void testMismatchedNNIsRejected() throws Exception {
           .manageNameDfsDirs(false).format(false).checkExitOnShutdown(false)
           .build();
       fail("New NN with different namespace should have been rejected");
-    } catch (ExitException ee) {
+    } catch (IOException ie) {
       GenericTestUtils.assertExceptionContains(
-          "Unable to start log segment 1: too few journals", ee);
-      assertTrue("Didn't terminate properly ", ExitUtil.terminateCalled());
+          "recoverUnfinalizedSegments failed for too many journals", ie);

Review Comment:
   I see you made this change by catching it in `FSNamesystem#loadFSImage()` but now I'm having second thoughts about it. The old behavior comes from this block in `FSEditLog#startLogSegment()`:
   https://github.com/apache/hadoop/blob/63db1a85e376c2266afdc62b9590e40acc98429c/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java#L1437-L1447
   Here we catch the `IOException` _within FSEditLog_ itself and add some useful contextual information, plus we clean up the `journalSet`. Seems useful. Given that `recoverUnclosedStreams()` is called from a variety of places, it seems that we can't just copy this approach by adding the `terminate(1)` logic within `FSEditLog#recoverUnclosedStreams()` -- e.g. `Checkpointer#doCheckpoint()` seems to assume it can catch an `IOException` from this phase and try again, and `NameNode#initializeSharedEdits()` / `NameNode#copyEditLogSegmentsToSharedDir()` have their own termination logic that we should probably respect. But it also doesn't really feel correct to me to add this special exception. What I am thinking now is to add a new flag to `recoverUnclosedStreams()`, like `boolean terminateOnFailure`, that we would set true in `FSNamesystem#startActiveServices()` and `FSImage#initEditLog()` and false for the three other callers I mentioned above. If `true`, we just rethrow the original `IO
 Exception`, but otherwise, we use similar logic as from `startLogSegment()` to add some additional context and call the termination. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org