[jira] [Comment Edited] (HDFS-12733) Option to disable to namenode local edits

["Hanisha Koneru (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222907#comment-16222907 ] 

Hanisha Koneru edited comment on HDFS-12733 at 10/27/17 10:04 PM:
------------------------------------------------------------------

Thanks for the patch [~brahmareddy]. 
I have a few comments:
* The two new test cases in _TestFailureOfSharedDir_ - {{testLocalEditsDisabledWhenRequiredEditsConfigured}} and {{testLocalEditsDisabled}} pass irrespective of the change in {{FSNamesystem}}.

* In _testLocalEditsDisabledWhenRequiredEditsConfigured_, we have the following configuration settings:
{code}
dfs.namenode.edits.dir            --> foo, bar
dfs.namenode.shared.edits.dir     --> bar
dfs.namenode.edits.dir.required   --> foo
{code}
Here, {{edits.dir.required}} is configured with a dir not present in {{shared.edits.dir}}
As per the test case, {{foo}} should be present in {{requiredEditDirs}}, as expected.
{code}
} else {
          assertTrue(
          Joiner.on(",").join(requiredEditsDirs) + " does not contain " + foo,
          requiredEditsDirs.contains(foo));
}
{code}
But this configuration would fail the {{FSNamesystem#checkConfiguration}} check. This function checks that all dirs in {{edits.dir.required}} are present in either {{shared.edits.dir}} or {{dfs.namenode.edits.dir}}. And since {{dfs.namenode.edits.dir}} is not included in {{getNamespaceEditsDirs}}, we get an exception.
{code}
java.lang.IllegalArgumentException: Required edits directory file:/foo not found: dfs.namenode.edits.dir=[file:/bar]; dfs.namenode.edits.dir.required=[file:/bar, file:/foo]; dfs.namenode.shared.edits.dir=[file:/bar]
{code}

So, a few options I can think of:
- the {{edits.dir.required}} should be a subset of {{shared.edits.dir}} when local edits are disabled. Add a check for this in {{FSNamesystem#checkConfiguration}}.
- local edits cannot be disabled if {{edits.dir.required}} has a dir present in {{edits.dir}} but not in {{shared.edits.dir}}
- {{FSNamesystem#getNamespaceEditsDirs}} adds {{edits.dir.required}} to the list of edit dirs even if it is not present in {{edits.dir}}. I am not sure this would be good idea though. Might break something else.


was (Author: hanishakoneru):
Thanks for the patch [~brahmareddy]. 
I have a few comments:
* The two new test cases in _TestFailureOfSharedDir_ - {{testLocalEditsDisabledWhenRequiredEditsConfigured}} and {{testLocalEditsDisabled}} pass irrespective of the change in {{FSNamesystem}}.

* In _testLocalEditsDisabledWhenRequiredEditsConfigured_, we have the following configuration settings:
{code}
dfs.namenode.edits.dir            --> foo, bar
dfs.namenode.shared.edits.dir     --> bar
dfs.namenode.edits.dir.required   --> foo
{code}
Here, {{edits.dir.required}} is configured with a dir not present in {{shared.edits.dir}}
As per the test case, {{foo}} should be present in {{requiredEditDirs}}, as expected.
{code}
} else {
          assertTrue(
          Joiner.on(",").join(requiredEditsDirs) + " does not contain " + foo,
          requiredEditsDirs.contains(foo));
}
{code}
But this configuration would fail the {{FSNamesystem#checkConfiguration}} check. This function checks that all dirs in {{edits.dir.required}} are present in either {{shared.edits.dir}} or {{dfs.namenode.edits.dir}}. And since {{dfs.namenode.edits.dir}} is not included in {{getNamespaceEditsDirs}}, we get an exception.
{code}
java.lang.IllegalArgumentException: Required edits directory file:/foo not found: dfs.namenode.edits.dir=[file:/bar]; dfs.namenode.edits.dir.required=[file:/bar, file:/foo]; dfs.namenode.shared.edits.dir=[file:/bar]
{code}

> Option to disable to namenode local edits
> -----------------------------------------
>
>                 Key: HDFS-12733
>                 URL: https://issues.apache.org/jira/browse/HDFS-12733
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Brahma Reddy Battula
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-12733-001.patch
>
>
> As of now, Edits will be written in local and shared locations which will be redundant and local edits never used in HA setup.
> Disabling local edits gives little performance improvement.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org