You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Szilard Nemeth (Jira)" <ji...@apache.org> on 2023/03/05 16:22:00 UTC

[jira] [Comment Edited] (YARN-11415) Refactor TestConfigurationFieldsBase and the connected test classes

    [ https://issues.apache.org/jira/browse/YARN-11415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696584#comment-17696584 ] 

Szilard Nemeth edited comment on YARN-11415 at 3/5/23 4:21 PM:
---------------------------------------------------------------

Hi [~bteke],
I just briefly checked this.
{quote}Another issue with this is that it works in a semi-opposite way of what it's supposed to do. To ensure all of the configuration entries are documented it should iterate through all of the configuration fields and check if those have matching xyz-default.xml entries, but currently it just reports the entries that are present in the xyz-default.xml and missing in the matching configuration file. Since this test checks all the configuration objects this might need some other follow-ups to document the missing properties from other components if there are any.
{quote}
I think what you stated here is true for this method: TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass.
This method compares the properties that are in yarn-default.xml, but not in the Configuration class.

*With my first commit* I added a dummy property to yarn-default.xml without adding it to the YarnConfiguration class.
The property called "yarn.nodemanager.missingpropinclass", but the name doesn't really matter.

Test result: Failure in TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass:
{code:java}
java.lang.AssertionError: yarn-default.xml has 1 properties missing in  class org.apache.hadoop.yarn.conf.YarnConfiguration Entries:   yarn.nodemanager.missingpropinclass 
Expected :0
Actual   :1
{code}
However, we also have TestCommonConfigurationFields#testCompareConfigurationClassAgainstXml which compares the properties that are in the YarnConfiguration class, but not defined in yarn-default.xml.
----
{*}So with my second commit{*}, I added this to YarnConfiguration:
{code:java}
public static final String MISSING_PROP_IN_YARN_DEF = "yarn.missingprop.in.yarndefault";
{code}
without touching the yarn-default.xml so the new config was not documented.

As I expected, the test case TestConfigurationFieldsBase#testCompareConfigurationClassAgainstXml failed with:
{code:java}
java.lang.AssertionError: class org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing in yarn-default.xml Entries:   yarn.missingprop.in.yarndefault 
Expected :0
Actual   :1
{code}
I think so far so good, this is the expected behavior.

The main issue before YARN-11413 was: 
1. org.apache.hadoop.conf.TestConfigurationFieldsBase#setupTestConfigurationFields is called as a "@Before" method.
2. org.apache.hadoop.conf.TestConfigurationFieldsBase#extractMemberVariablesFromConfigurationFields is called.
3. All of the fields of the class is checked here with certain restrictions. Among these are that it should be a public static final String, and it should match a Pattern.
If the pattern is not matched, the field won't be added to the "known fields" for sure.
----
 

{*}4. So with my third commit{*}, I just removed the percent sign (basically a revert of YARN-11415) to see what happens.
TestConfigurationFieldsBase#testCompareXmlAgainstConfigurationClass fails, which is a false positive now.

Here's the assertion message:
{code:java}
java.lang.AssertionError: yarn-default.xml has 2 properties missing in  class org.apache.hadoop.yarn.conf.YarnConfiguration Entries:   yarn.nodemanager.aux-services.%s.classpath  yarn.nodemanager.aux-services.%s.system-classes 
Expected :0
Actual   :2
{code}
This is indeed wrong, as per your description.
However, I don't see how this could be fixed in a clean way.
Here, the configuration fields of YarnConfiguration were not recognized: yarn.nodemanager.aux-services.%s.classpath yarn.nodemanager.aux-services.%s.system-classes.
What can we do then? 
----
*CONCLUSION*

The whole point of matching the values of the String fields is to differentiate config keys from other strings like:
{code:java}
public static final String NVIDIA_DOCKER_V1 = "nvidia-docker-v1";
{code}
I cannot see a better way than what we are currently doing with the regex.
I think the regex pattern should be continuously maintained and occassions like unmatched real config keys like "yarn.nodemanager.aux-services.%s.classpath" (because of the regex didn't contain the percentage sign) could be quite rare.

[~bteke]
Do you have anything in mind about a clean fix for this?

Anyway, I will report another jira where I will list my suggested improvements of the test and its base class: TestConfigurationFieldsBase.


was (Author: snemeth):
Hi [~bteke],
I just briefly checked this.

{quote}
Another issue with this is that it works in a semi-opposite way of what it's supposed to do. To ensure all of the configuration entries are documented it should iterate through all of the configuration fields and check if those have matching xyz-default.xml entries, but currently it just reports the entries that are present in the xyz-default.xml and missing in the matching configuration file. Since this test checks all the configuration objects this might need some other follow-ups to document the missing properties from other components if there are any.
{quote}

I think what you stated here is true for this method: TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass.
This method compares the properties that are in yarn-default.xml, but not in the Configuration class.
With my first commit I added a dummy property to yarn-default.xml without adding it to the YarnConfiguration class.
The property called "yarn.nodemanager.missingpropinclass", but the name doesn't really matter.

Test result: Failure in TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass: 
{code}
java.lang.AssertionError: yarn-default.xml has 1 properties missing in  class org.apache.hadoop.yarn.conf.YarnConfiguration Entries:   yarn.nodemanager.missingpropinclass 
Expected :0
Actual   :1
{code}


However, we also have TestCommonConfigurationFields#testCompareConfigurationClassAgainstXml which compares the properties that are in the YarnConfiguration class, but not defined in yarn-default.xml.
So with my second commit, I added this to YarnConfiguration: 

{code}
public static final String MISSING_PROP_IN_YARN_DEF = "yarn.missingprop.in.yarndefault";
{code}

without touching the yarn-default.xml so the new config was not documented.

As I expected, the testcase TestConfigurationFieldsBase#testCompareConfigurationClassAgainstXml failed with: 
{code}
java.lang.AssertionError: class org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing in yarn-default.xml Entries:   yarn.missingprop.in.yarndefault 
Expected :0
Actual   :1
{code}

I think so far so good, this is the expected behavior.


The main issue before YARN-11413 was: 
1. org.apache.hadoop.conf.TestConfigurationFieldsBase#setupTestConfigurationFields is called as a "\@Before" method.
2. org.apache.hadoop.conf.TestConfigurationFieldsBase#extractMemberVariablesFromConfigurationFields is called.
3. All of the fields of the class is checked here with certain restrictions. Among these are that it should be a public static final String, and it should match a Pattern.
If the pattern is not matched, the field won't be added to the "known fields" for sure.


4. So with my third commit, I just removed the percent sign (basically a revert of YARN-11415) to see what happens.
TestConfigurationFieldsBase#testCompareXmlAgainstConfigurationClass fails, which is a false positive now.

Here's the assertion message: 
{code}
java.lang.AssertionError: yarn-default.xml has 2 properties missing in  class org.apache.hadoop.yarn.conf.YarnConfiguration Entries:   yarn.nodemanager.aux-services.%s.classpath  yarn.nodemanager.aux-services.%s.system-classes 
Expected :0
Actual   :2
{code}

This is indeed wrong, as per your description.
However, I don't see how this could be fixed in a clean way.
Here, the configuration fields of YarnConfiguration were not recognized: yarn.nodemanager.aux-services.%s.classpath  yarn.nodemanager.aux-services.%s.system-classes.
What can we do then? 
The whole point of matching the values of the String fields is to differentiate config keys from other strings like: 

{code}
public static final String NVIDIA_DOCKER_V1 = "nvidia-docker-v1";
{code}

I cannot see a better way than what we are currently doing with the regex.
I think the regex pattern should be continuously maintained and occassions like unmatched real config keys like "yarn.nodemanager.aux-services.%s.classpath" (because of the regex didn't contain the percentage sign) could be quite rare.

[~bteke]
Do you have anything in mind about a clean fix for this? 

Anyway, I will report another jira where I will list my suggested improvements of the test and its base class: TestConfigurationFieldsBase.



> Refactor TestConfigurationFieldsBase and the connected test classes
> -------------------------------------------------------------------
>
>                 Key: YARN-11415
>                 URL: https://issues.apache.org/jira/browse/YARN-11415
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Benjamin Teke
>            Assignee: Szilard Nemeth
>            Priority: Major
>              Labels: pull-request-available
>
> YARN-11413 pointed out a strange way of how the configuration tests are executed. The first problem is that there is a [Pattern|https://github.com/apache/hadoop/blob/570b503e3e7e7adf5b0a8fabca76003298216543/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfigurationFieldsBase.java#L197], that matches only numbers, letters, dots, hyphens and underscores, but not %, which is used in string replacements (i.e {{yarn.nodemanager.aux-services.%s.classpath}} ), so essentially every property that's present in any configuration object and doesn't match this pattern is silently skipped, and documenting it will result in invalid test failures, ergo the test encourages introducing props and not documenting them. The pattern should be fixed in YARN-11413 for %s, but it's necessity could be checked. 
> Another issue with this is that it works in a semi-opposite way of what it's supposed to do. To ensure all of the configuration entries are documented it should iterate through all of the configuration fields and check if those have matching xyz-default.xml entries, but currently it just reports the entries that are present in the xyz-default.xml and missing in the matching configuration file. Since this test checks all the configuration objects this might need some other follow-ups to document the missing properties from other components if there are any.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org