You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "steven zhuang (JIRA)" <ji...@apache.org> on 2011/06/24 04:05:47 UTC
[jira] [Created] (HADOOP-7425) ReflectionUtils.setConf would
configure the KeyFieldBasedPartitioner twice in Hadoop 0.21.0, when
KeyFieldBasedPartitioner is an Configurable instance
ReflectionUtils.setConf would configure the KeyFieldBasedPartitioner twice in Hadoop 0.21.0, when KeyFieldBasedPartitioner is an Configurable instance
------------------------------------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-7425
URL: https://issues.apache.org/jira/browse/HADOOP-7425
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 0.21.0
Reporter: steven zhuang
In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
public static void setConf(Object theObject, Configuration conf) {
if (conf != null) {
if (theObject instanceof Configurable) {
((Configurable) theObject).setConf(conf);
}
setJobConf(theObject, conf);
}
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: Hadoop-7425.patch
Made changes to set configuration only once for the old/new api
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: Hadoop-7425.patch
changes to apply configuration only once
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
steven zhuang updated HADOOP-7425:
----------------------------------
Attachment: hadoop7425.patch
this file contains patch to ReflectionUtils.java and TestReflectionUtils.java.
setConf is a method called many times, this patch may have profound influence if it introduces any bug.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.2, 0.20.205.0, 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, hadoop7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064305#comment-13064305 ]
steven zhuang commented on HADOOP-7425:
---------------------------------------
hi, Sudharsan, sure, you can use the data and scripts I uploaded(later) to do the check.
I used the "org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner" instead of the newer "org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedPartitioner" in streaming command cause in 0.21.0, the org.apache.hadoop.mapred.JobConf.setPartitionerClass() is still expecting an Object extends from "org.apache.hadoop.mapred.Partitioner":
public void setPartitionerClass(Class<? extends Partitioner> theClass) {
' setClass("mapred.partitioner.class", theClass, Partitioner.class);
}
and why is KeyFieldBasedPartitioner configured twice here:
cause the "o.a.h.mapred.lib.KeyFieldBasedPartitioner" class now extends from "o.a.h.mapreduce.lib.partition.KeyFieldBasedPartitioner" to use potential newer features(which's not appeared yet), which is Configurable, so in ReflectionUtils.setConf, the line :
"if (theObject instanceof Configurable) { ((Configurable) theObject).setConf(conf); }"
will be executed, configure the partitioner once, add a KeyDescription to the KeyDescription list.
And later, no matter the above line is executed or what, this line:
"setJobConf(theObject, conf);"
will be executed anyway, configure the Partitioner twice, add another KeyDescription to the list, even if we just configured one.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0
> Reporter: steven zhuang
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218477#comment-13218477 ]
Robert Joseph Evans commented on HADOOP-7425:
---------------------------------------------
Steven,
The build system on trunk has changed to use maven instead of ant so the location of the files are different. In addition to this common, mapreduce, and HDFS were split up and common no longer has any direct dependencies on mapreduce, but the new test does, so it will not compile. ReflectionUtils loads the mapreduce dependencies using reflection now, which is ugly, but works. If you want this fix to go into the 1.0 line, 0.21 or even 0.22 we also need to have a version that is compatible with trunk so we do not get any regressions in newer versions.
In addition to this, looking at the code I am not sure of all of the ramifications of not having JobConfiguration.configure called for all classes that are both JobConfigurable and Configurable. There are other classes that may rely on this behavior. I think I would prefer to see KeyFieldBasedPartitioner updated to handle having both methods called.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.20.2, 0.20.205.0, 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, hadoop7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067538#comment-13067538 ]
Hadoop QA commented on HADOOP-7425:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12486973/Hadoop-7425.patch
against trunk revision 1147971.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//console
This message is automatically generated.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated HADOOP-7425:
----------------------------------------
Status: Open (was: Patch Available)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.23.0, 0.21.0, 0.20.205.0, 0.20.2
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, hadoop7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064633#comment-13064633 ]
steven zhuang commented on HADOOP-7425:
---------------------------------------
hi Sudharsan,
I think the KeyFieldHelper has done its job, it is the setConf method of ReflectionUtils makes the partitioner configured twice.
I wonder why the logic in ReflectionUtils.setConf first call setConf() then call configure() on the same object? can we write it as:
if (theObject instanceof Configurable) {
((Configurable) theObject).setConf(conf);
}
else {
setJobConf(theObject, conf);
}
can we do this?
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: (was: Hadoop-7425.patch)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
steven zhuang updated HADOOP-7425:
----------------------------------
Attachment: test.tar
use the scripts in this tar ball to test the KeyFieldBasedPartitioner.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0
> Reporter: steven zhuang
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063869#comment-13063869 ]
Sudharsan Sampath commented on HADOOP-7425:
-------------------------------------------
can you provide a sample code that cause you the issue
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0
> Reporter: steven zhuang
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064494#comment-13064494 ]
Sudharsan Sampath commented on HADOOP-7425:
-------------------------------------------
Hi Steven,
Yes. You are right. This issue is seen on the trunk too. But to me, configuring twice should not be an issue but processing the configured property without realizing that the same configuration has been processed already could be avoided.
Will it be ok if instead of maintaining a List, we maintain a Set of keyspecs in the KeyFieldHelper and provide an equals method that returns true if its params match exactly as the passed in. The downside is that if some job wants this kind of double hashing done on the same keyspec that would be difficult asking for a custom partitioner. But I doubt if there would be such a scenario.
This would also ensure we get the correct partition and increase performance as we avoid a hash invoke for every record.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067769#comment-13067769 ]
steven zhuang commented on HADOOP-7425:
---------------------------------------
thanks Sudharsan, you are doing great job here. will we see this modification in new release of hadoop?
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
steven zhuang updated HADOOP-7425:
----------------------------------
Description:
In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
public static void setConf(Object theObject, Configuration conf) {
if (conf != null) {
if (theObject instanceof Configurable) {
((Configurable) theObject).setConf(conf);
}
setJobConf(theObject, conf);
}
}
was:
In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
public static void setConf(Object theObject, Configuration conf) {
if (conf != null) {
if (theObject instanceof Configurable) {
((Configurable) theObject).setConf(conf);
}
setJobConf(theObject, conf);
}
}
Summary: ReflectionUtils.setConf would configure anything Configurable twice (was: ReflectionUtils.setConf would configure the KeyFieldBasedPartitioner twice in Hadoop 0.21.0, when KeyFieldBasedPartitioner is an Configurable instance)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0
> Reporter: steven zhuang
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: Hadoop-7425.patch
Changes to set configuration only once for the new/old api
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: (was: HADOOP-7425.patch)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: HADOOP-7425.patch
Added test case to the patch
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated HADOOP-7425:
----------------------------------------
Status: Open (was: Patch Available)
The latest test patch failed. Also the patch is rather old, and will need to be upmerged to apply to trunk.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.23.0, 0.21.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: (was: Hadoop-7425.patch)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: (was: Hadoop-7425.patch)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Affects Version/s: 0.23.0
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Attachments: test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "steven zhuang (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
steven zhuang updated HADOOP-7425:
----------------------------------
Affects Version/s: 0.20.2
0.20.205.0
Status: Patch Available (was: Open)
so glad this problem is updated.
and Robert, I think the given patch might not point out how the problem is gonna happen. the tested object should be an object from a class implements these two interfaces: Configurable, JobConfigurable, only in this case will the object be configured twice(1. setConf and 2. configure).
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.23.0, 0.21.0, 0.20.205.0, 0.20.2
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068178#comment-13068178 ]
Hadoop QA commented on HADOOP-7425:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12487111/HADOOP-7425.patch
against trunk revision 1147971.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
-1 system test framework. The patch failed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/752//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/752//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/752//console
This message is automatically generated.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Tags: ReflectionUtils
Labels: patch (was: )
Status: Patch Available (was: Open)
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: Hadoop-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
configure anything Configurable twice
Posted by "Sudharsan Sampath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudharsan Sampath updated HADOOP-7425:
--------------------------------------
Attachment: HADOOP-7425.patch
Updated with proper test case. We should add a test case on JobConfigurable or KeyFieldBasedPartitioner on their respective test classes.
> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
> Key: HADOOP-7425
> URL: https://issues.apache.org/jira/browse/HADOOP-7425
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 0.21.0, 0.23.0
> Reporter: steven zhuang
> Labels: patch
> Attachments: HADOOP-7425.patch, test.tar
>
>
> In the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number.
> public static void setConf(Object theObject, Configuration conf) {
> if (conf != null) {
> if (theObject instanceof Configurable) {
> ((Configurable) theObject).setConf(conf);
> }
> setJobConf(theObject, conf);
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira