You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2010/07/14 02:39:53 UTC
[jira] Created: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Ability for having user's classes take precedence over the system classes for tasks' classpath
----------------------------------------------------------------------------------------------
Key: MAPREDUCE-1938
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: job submission, task, tasktracker
Reporter: Devaraj Das
Fix For: 0.22.0
It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated MAPREDUCE-1938:
-----------------------------------
Attachment: mr-1938-bp20.patch
Patch for Y20. Not for commit here.
The patch defines a configuration option using which users can specify that the system classpath be added last in the list of classpaths created before launching a task. It also introduces a HADOOP_CLIENT_CLASSPATH env variable in the hadoop shell script that should be set to point to the user's jar/classpath (containing his implementation of the system classes) before submitting a job over command line.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888482#action_12888482 ]
Doug Cutting commented on MAPREDUCE-1938:
-----------------------------------------
Owen, I agree with your analysis. I'm just trying to put this patch in context of these other related discussions.
This patch addresses some issues relevant to separation of kernel & library. In common cases one can merely provide an alternate version of the library class in one's job. Fully separating kernel & library with a well-defined, minimal kernel API is clearly aesthetically better. Are there use cases that will that enable that this patch will not? I think mostly it will just make it clear which classes are safe to replace with updated versions and which are not. Does that sound right?
The issue of user versions of libraries that the kernel uses (like Avro, log4j, HttpClient, etc.) is not entirely addressed by this patch. If the user's version is backwards compatible with the kernel's version then this patch is sufficient. But if the user's version of a library makes incompatible changes then we'd need a classloader/OSGI solution. Even then, I think it only works if user and kernel code do not interchange instances of classes defined by these libraries. A minimal kernel API will help reduce that risk. Does this analysis sound right?
I'm trying to understand how far this patch gets us towards those goals: what it solves and what it doesn't.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888445#action_12888445 ]
Owen O'Malley commented on MAPREDUCE-1938:
------------------------------------------
Doug,
I agree that the kernel code should be split out from libraries, however, that work is much more involved. I don't see a problem with putting the user's code first. It is not a security concern. The user's code is only run as the user. Furthermore, it doesn't actually stop them from loading system classes. They can exec a new jvm with a new class path of their own choosing.
Therefore, by putting the user's classes last all that we've done is make it harder for the user to implement hot fixes in their own jobs. That doesn't seem like a good goal.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "David Rosenstrauch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898259#action_12898259 ]
David Rosenstrauch commented on MAPREDUCE-1938:
-----------------------------------------------
Just wondering: does anyone know if Cloudera has plans to include this patch in the CDH any time soon?
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888536#action_12888536 ]
Owen O'Malley commented on MAPREDUCE-1938:
------------------------------------------
This patch basically puts the user in charge of their job. They can leave the safety switch set in which case they get the current behavior. But if they turn off the safety, their classes go ahead of the ones installed on the cluster. That means that they can break things, but all they can break is their own tasks.
After we do the split of core from library, you still need this switch. There will always be the possibility of needing to patch something in the core, because even MapTask has bugs. *smile* After splitting them apart, we can put the library code at the very end
safety on: core, user, library
safety off: user, core, library
This patch is just about providing the safety switch.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900388#action_12900388 ]
Luke Lu commented on MAPREDUCE-1938:
------------------------------------
A couple of questions regarding the patch (filename should probably end with .patch):
# Are the new property a system/cluster property or a per job property? i.e, allowing user to to override the behavior (default to system jars first) per job. Owen's comments seem to imply it's a per job switch, which should be defined/documented in MRJobConfig (MAPREDUCE-1749)
# Why there are two boolean properties defined in the trunk patch "mapreduce.user.classpath.first" (defined in TaskRunner) and "mapreduce.task.classpath.user.precedence" (defined in JobContext)? The latter seems to have no effect and not tested.
# Why use Vector (in the test code) when there doesn't seem to be need for the list to be synchronized?
# Why do the test code use hard code paths and string literals when there are corresponding constants defined?
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated MAPREDUCE-1938:
-----------------------------------
Attachment: mr-1938-bp20.1.patch
Addressing Owen's comment on the shell script part of the patch.
Doug, this patch is a first step towards letting users use their own versions of library provided implementation for things like CombineFileInputFormat. The use case is to allow for specific implementations of library classes for certain classes of jobs.
This doesn't aim to address the kernel/library separation in its entirety. So yes, if the user puts a class on the classpath that doesn't work with the kernel compatibly then tasks will fail, or produce obscure/inconsistent results, but that will affect only that job, and the user would notice that soon (hopefully). Did i understand your concern right?
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902586#action_12902586 ]
Krishna Ramachandran commented on MAPREDUCE-1938:
-------------------------------------------------
testpatch output
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888433#action_12888433 ]
Doug Cutting commented on MAPREDUCE-1938:
-----------------------------------------
Two thoughts:
1. In general, we need to better separate the kernel from the library. CombineFileInputFormat is library code and should be easy to update without updating the cluster. Long-term, only kernel code should be hardwired on the classpath of tasks, with library and user code both specified per job. There should be no default version of library classes for a task: tasks should always specify their required libraries. Is there a Jira for this? I know Tom's expressed interest in working on this.
2. We should permit user code to depend on different versions of things than the kernel does. For example, user code might rely on a different version of HttpClient or Avro than that used by MapReduce. This should be possible if instances of classes from these are not a passed between user and kernel code, e.g., as long as Avro and HttpClient classes are not a part of the MapReduce API. In this case classloaders (probably via OSGI) could permit this.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888532#action_12888532 ]
Doug Cutting commented on MAPREDUCE-1938:
-----------------------------------------
> Did i understand your concern right?
I don't have specific concerns about this patch. Sorry for any confusion in that regard. I thought it worthwhile to discuss how this change relates to other changes that are contemplated. It seems not inconsistent, provides some of the benefits, and is considerably simpler; in short, a good thing.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1938:
--------------------------------------------
Status: Patch Available (was: Open)
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1938:
--------------------------------------------
Attachment: mapred-1938-2.patch
Luke
Last one was a preliminary/quick port of what was there and lot of changes were redundant. I have cleaned it up in this rev.
As for moving to MRJobConfig, based on discussions, my understanding is, desired to have this flag semi-private (MRJobConfig i believe is very public)
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1938:
--------------------------------------------
Attachment: mapred-1938-1
We were going thro' major outstanding issues in the trunk
Thought will upload a preliminary port of Devaraj's patch to trunk + a simple test case
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900792#action_12900792 ]
Luke Lu commented on MAPREDUCE-1938:
------------------------------------
Nits in the test code aside, +1
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Krishna Ramachandran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Ramachandran updated MAPREDUCE-1938:
--------------------------------------------
Attachment: mapred-1938-3.patch
revised per previous comment
+ Luke's test case review - vector is API requirement - can't modify
Yes there will be a separate jira for bin/hadoop client command line changes
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Assignee: Krishna Ramachandran
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mapred-1938-3.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu reassigned MAPREDUCE-1938:
--------------------------------------------------
Assignee: Krishna Ramachandran
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Assignee: Krishna Ramachandran
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1938) Ability for having user's
classes take precedence over the system classes for tasks' classpath
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888436#action_12888436 ]
Owen O'Malley commented on MAPREDUCE-1938:
------------------------------------------
I think that the default for this should be on.
Rather than add HADOOP_CLIENT_CLASSPATH, let's make a new variable HADOOP_USER_CLASSPATH_LAST. If it is defined, we add HADOOP_CLASSPATH to the tail like we currently do. Otherwise it is added to the front.
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1938) Ability for having user's classes
take precedence over the system classes for tasks' classpath
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated MAPREDUCE-1938:
-----------------------------------------------
Status: Open (was: Patch Available)
bq. As for moving to MRJobConfig, based on discussions, my understanding is, desired to have this flag semi-private (MRJobConfig i believe is very public).
MRJobConfig is the place to hold all job-level config paramaters. And it is not public. Annotation says "private, evolving". So, the configuration should be in MRJobConfig. Also, the name should say mapreduce.job.user.classpath.first saying it is a job level property.
Also, in your next patch, can you do the changes in test-case suggested by Luke?
Are you planning to raise a Hadoop-common jira for the changes in bin/hadoop script?
> Ability for having user's classes take precedence over the system classes for tasks' classpath
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: job submission, task, tasktracker
> Reporter: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: mapred-1938-1, mapred-1938-2.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch
>
>
> It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.