You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Jim Donofrio (JIRA)" <ji...@apache.org> on 2012/05/30 04:26:22 UTC
[jira] [Created] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Jim Donofrio created MRUNIT-114:
-----------------------------------
Summary: should support calling configure on sorting and grouping comparators if applicable
Key: MRUNIT-114
URL: https://issues.apache.org/jira/browse/MRUNIT-114
Project: MRUnit
Issue Type: Bug
Reporter: Jim Donofrio
Assignee: Jim Donofrio
Fix For: 1.0.0
>From Jacob Metcalf:
{noformat}
On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
I am writing my own multiple input MR driver to test a reduce side join and had to do:
// Configure grouping and sorting comparators
if (keyGroupComparator instanceof Configured ) {
((Configured)keyGroupComparator).setConf( configuration );
}
if (keyValueOrderComparator instanceof Configured) {
((Configured)keyValueOrderComparator).setConf( configuration );
}
To get the config applied even though they are Configured objects.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio resolved MRUNIT-114.
---------------------------------
Resolution: Fixed
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466513#comment-13466513 ]
Jim Donofrio commented on MRUNIT-114:
-------------------------------------
Dave,
1. I will create a new JIRA.
2. I did not believe that this is a problem, if you are using the mapreduce api then you should be using the mapreduce org.apache.hadoop.mapreduce.lib.partition.KeyFieldBasedComparator which implements Configurable
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466698#comment-13466698 ]
Dave Beech commented on MRUNIT-114:
-----------------------------------
Cheers Jim. I wasn't aware of the new mapreduce api KeyFieldBasedComparator - thanks for pointing that out. I think the problem still applies to the old mapred API though so I'll create a new JIRA for this too. I agree that none of these issues should block 1.0.0.
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466515#comment-13466515 ]
Jim Donofrio commented on MRUNIT-114:
-------------------------------------
I do not believe that #1 should be a blocker for 1.0.0
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294522#comment-13294522 ]
Dave Beech commented on MRUNIT-114:
-----------------------------------
Hi Jim - I'm working with the KeyFieldBasedComparator at the moment and I've found a couple more issues you might need to consider.
1. Because the shuffle method in MapReduceDriverBase calls the key ordering comparator using compare(Object,Object), it does not call the correct method on classes which only override and do all the work in compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2), like KeyFieldBasedComparator.
It might be better to check whether the two keys are BinaryComparable, casting, and calling this method if possible. Even if a user's comparator doesn't override it, the default implementation of this in WritableComparator would still be OK.
2. KeyFieldBasedComparator doesn't get configured properly even after the change in 1344239, because ReflectionUtils.newInstance requires the provided config to be a JobConf in order for it to call the configure method on a JobConfigurable object. Of course, the trouble is that simply wrapping getConfiguration as new JobConf(getConfiguration()) isn't a great thing to do as MapReduceDriverBase is shared by the mapreduce api version...
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285642#comment-13285642 ]
Jim Donofrio commented on MRUNIT-114:
-------------------------------------
Jacob,
You should not have to call setConf anymore
checked in 1344239
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470973#comment-13470973 ]
Dave Beech commented on MRUNIT-114:
-----------------------------------
Created MRUNIT-155 for this second issue
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466516#comment-13466516 ]
Jim Donofrio commented on MRUNIT-114:
-------------------------------------
created MRUNIT-154 for issue #1
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-114:
--------------------------------
Description:
>From Jacob Metcalf:
On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
I am writing my own multiple input MR driver to test a reduce side join and had to do:
// Configure grouping and sorting comparators
if (keyGroupComparator instanceof Configured ) {
((Configured)keyGroupComparator).setConf( configuration );
}
if (keyValueOrderComparator instanceof Configured) {
((Configured)keyValueOrderComparator).setConf( configuration );
}
To get the config applied even though they are Configured objects.
was:
>From Jacob Metcalf:
{noformat}
On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
I am writing my own multiple input MR driver to test a reduce side join and had to do:
// Configure grouping and sorting comparators
if (keyGroupComparator instanceof Configured ) {
((Configured)keyGroupComparator).setConf( configuration );
}
if (keyValueOrderComparator instanceof Configured) {
((Configured)keyValueOrderComparator).setConf( configuration );
}
To get the config applied even though they are Configured objects.
{noformat}
We fail to use ReflectionUtils.newInstance on the 2 comparators, it would be better if most of these classes only took classes as input instead of the actual object
{noformat}
public void setKeyGroupingComparator(
final RawComparator<K2> groupingComparator) {
keyGroupComparator = ReflectionUtils.newInstance(returnNonNull(groupingComparator).getClass(), getConfiguration());
}
public void setKeyOrderComparator(final RawComparator<K2> orderComparator) {
keyValueOrderComparator = ReflectionUtils.newInstance(returnNonNull(orderComparator), getConfiguration());
}
{noformat}
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466828#comment-13466828 ]
Jim Donofrio commented on MRUNIT-114:
-------------------------------------
Just to clarify, are you saying that the mapred KeyFieldBasedComparator will only get configured with a JobConf object and not a Configuration object? I guess we could work around that but hadoop will act the same way but if you are using mapred, hadoop forces you to submit JobConf at some point, maybe we should do the same?
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-114:
--------------------------------
Reporter: Jacob Metcalf (was: Jim Donofrio)
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466844#comment-13466844 ]
Dave Beech commented on MRUNIT-114:
-----------------------------------
That's correct, because the mapred api KeyFieldBasedComparator is JobConfigurable instead of Configurable. It will only be configured with a JobConf, not a plain Configuration.
When I first noticed this I was hesitant to use JobConf anywhere for the new mapreduce API version of our test runners, but from looking through Hadoop source I see that JobConf is actually used pretty much everywhere internally. It's just no longer used by the client when setting up a job.
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285348#comment-13285348 ]
Jim Donofrio commented on MRUNIT-114:
-------------------------------------
I think in the future we should get rid of all of these etKeyGroupingComparator type of methods and solely use the jobconf
Any thoughts?
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-114) should support calling configure on
sorting and grouping comparators if applicable
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294533#comment-13294533 ]
Dave Beech commented on MRUNIT-114:
-----------------------------------
This one just caught me out also. Because of the fact that ReflectionUtils.newInstance is called at the time you call the setter, rather than at test run time, it's important to call set/withConfiguration *before* calling set/withKeyGroupingComparator or set/withKeyOrderingComparator, otherwise the custom config parameters won't get used.
> should support calling configure on sorting and grouping comparators if applicable
> ----------------------------------------------------------------------------------
>
> Key: MRUNIT-114
> URL: https://issues.apache.org/jira/browse/MRUNIT-114
> Project: MRUnit
> Issue Type: Bug
> Reporter: Jacob Metcalf
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From Jacob Metcalf:
> On another subject (maybe I need a new thread/JIRA for this) is it intentional that the configuration is not applied to the sorting and grouping comparators?
> I am writing my own multiple input MR driver to test a reduce side join and had to do:
> // Configure grouping and sorting comparators
> if (keyGroupComparator instanceof Configured ) {
> ((Configured)keyGroupComparator).setConf( configuration );
> }
> if (keyValueOrderComparator instanceof Configured) {
> ((Configured)keyValueOrderComparator).setConf( configuration );
> }
> To get the config applied even though they are Configured objects.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira