You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Dennis Kubes (JIRA)" <ji...@apache.org> on 2008/04/15 22:06:22 UTC
[jira] Created: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Allow Mapper and Reducer classes to be created a Spring Beans
-------------------------------------------------------------
Key: HADOOP-3261
URL: https://issues.apache.org/jira/browse/HADOOP-3261
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Environment: All
Reporter: Dennis Kubes
Assignee: Dennis Kubes
This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590579#action_12590579 ]
Dennis Kubes commented on HADOOP-3261:
--------------------------------------
For one, the next generation of nutch would benefit from this. If we were to remove the plugin architecture we could use dependency injection in its place which would make it easier to change out components and to test various component pieces.
Right now most of the objects are created through ReflectionUtils. Should be easy to change that out with some type of factory class which allow object to be created via Reflection, Spring, something else. It should work for partitioners and the others as well.
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12590537#action_12590537 ]
Doug Cutting commented on HADOOP-3261:
--------------------------------------
What applications would benefit from this?
Also, wouldn't we want to implement this for partitioners, inputformats, outputformats, comparators, etc. too?
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591442#action_12591442 ]
Dennis Kubes commented on HADOOP-3261:
--------------------------------------
Sure, I will make the changes. I just wanted to give an example of possibility before and I saw that ReflectionUtils was used all over the codebase and didn't know if having a DI creator in that class would have unintended side effects. One other thing. We may not want to limit this to just spring. The ability to create these classes and others could be handled in a number of different ways by different frameworks, spring, guice, etc. I think it is something to think about.
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12591441#action_12591441 ]
Owen O'Malley commented on HADOOP-3261:
---------------------------------------
Rather than making new factories, wouldn't it be better to make ReflectionUtils.newInstance do all of the work? That way all of the contexts will get it automatically. Does that make sense?
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dennis Kubes updated HADOOP-3261:
---------------------------------
Attachment: HADOOP-3261-1-20080415.patch
Patch that allows Mappers and Reducer classes to be created as spring beans and have their dependencies injected. An example spring job is included.
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-3261.
-----------------------------------
Resolution: Won't Fix
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592122#action_12592122 ]
Dennis Kubes commented on HADOOP-3261:
--------------------------------------
After talking with doug and doing a little more prototyping, this really isn't needed. We can get the same benefit (in nutch) simply by using the configuration setup like hadoop does :) with reflectionutils instead of the plugin architecture. IMO this issue can be closed.
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: HADOOP-3261-1-20080415.patch, spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3261) Allow Mapper and Reducer classes to
be created a Spring Beans
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dennis Kubes updated HADOOP-3261:
---------------------------------
Attachment: spring.jar
Spring 2.5.3 jar file, licensed under an apache license.
> Allow Mapper and Reducer classes to be created a Spring Beans
> -------------------------------------------------------------
>
> Key: HADOOP-3261
> URL: https://issues.apache.org/jira/browse/HADOOP-3261
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Environment: All
> Reporter: Dennis Kubes
> Assignee: Dennis Kubes
> Attachments: spring.jar
>
>
> This improvement abstracts the creation of Mappers and Reducers in factory classes and adds the ability to define a spring beans.xml file in the classpath and have the Mapper and Reducer classes be created as spring beans. This is transparent to the rest of Hadoop but the benefit is that is allows Mapper and Reducer classes to have their dependencies injected versus having them hardcoded. This will help in the ability to test various mapreduce jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.