You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/11/03 16:42:00 UTC
[jira] [Comment Edited] (HADOOP-15015) TestConfigurationFieldsBase
to use SLF4J for logging
[ https://issues.apache.org/jira/browse/HADOOP-15015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237876#comment-16237876 ]
Steve Loughran edited comment on HADOOP-15015 at 11/3/17 4:41 PM:
------------------------------------------------------------------
Patch 001
# main log for normal output. special xml and config ones for debugging at that level.
# moved all the logging
# Deleted the old boolean keys...nothing downstream was using them.
# Added empty sets to the relevant fields, so we can eliminate all the != null checks
# Sort things for output by feeding sets through TreeSet before iterating through the values.
# Assertion errors now include error text.
# Went with the IDE's java 8 migrations to leaner code all round, except for the most trivial for loops
Example
{code}
Failed tests:
TestMapreduceConfigFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass:542 mapred-default.xml has 2 properties missing in interface org.apache.hadoop.mapreduce.MRJobConfig interface org.apache.hadoop.mapreduce.MRConfig class org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig class org.apache.hadoop.mapred.ShuffleHandler class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat class org.apache.hadoop.mapreduce.lib.input.FileInputFormat class org.apache.hadoop.mapreduce.Job class org.apache.hadoop.mapreduce.lib.input.NLineInputFormat class org.apache.hadoop.mapred.JobConf class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter Entries: mapreduce.outputcommitter.factory.class mapreduce.outputcommitter.factory.scheme.s3a expected:<0> but was:<2>
{code}
Getting that list of missing entries is key, as now Jenkins will include its list of problems in all its failed tests. You shouldn't need to look at the output logs to debug basic problems.
was (Author: stevel@apache.org):
Patch 001
1. main log for normal output. special xml and config ones for debugging at that level.
1. moved all the logging
1. Deleted the old boolean keys...nothing downstream was using them.
1. Added empty sets to the relevant fields, so we can eliminate all the != null checks
1. Sort things for output by feeding sets through TreeSet before iterating through the values.
1. Assertion errors now include error text.
1. Went with the IDE's java 8 migrations to leaner code all round
Example
{code}
Failed tests:
TestMapreduceConfigFields>TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass:542 mapred-default.xml has 2 properties missing in interface org.apache.hadoop.mapreduce.MRJobConfig interface org.apache.hadoop.mapreduce.MRConfig class org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig class org.apache.hadoop.mapred.ShuffleHandler class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat class org.apache.hadoop.mapreduce.lib.input.FileInputFormat class org.apache.hadoop.mapreduce.Job class org.apache.hadoop.mapreduce.lib.input.NLineInputFormat class org.apache.hadoop.mapred.JobConf class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter Entries: mapreduce.outputcommitter.factory.class mapreduce.outputcommitter.factory.scheme.s3a expected:<0> but was:<2>
{code}
Getting that list of missing entries is key, as now Jenkins will include its list of problems in all its failed tests. You shouldn't need to look at the output logs to debug basic problems.
> TestConfigurationFieldsBase to use SLF4J for logging
> ----------------------------------------------------
>
> Key: HADOOP-15015
> URL: https://issues.apache.org/jira/browse/HADOOP-15015
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf, test
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-15015-001.patch
>
>
> {{TestConfigurationFieldsBase}} has a protected "configDebug" field used to turn logging on/off
> {code}
> if (configDebug) {
> System.out.println("Field: " + f);
> }
> {code}
> Presumably its there to allow people with code access to debug their classes. But if we switch to SLF4J you get controllable logging @ runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org