[jira] [Commented] (MAPREDUCE-5146) application classloader may be used too early to load classes

["Sangjin Lee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13629341#comment-13629341 ] 

Sangjin Lee commented on MAPREDUCE-5146:
----------------------------------------

Example stack trace:

{noformat}
	org.apache.hadoop.yarn.util.ApplicationClassLoader.loadClass(java.lang.String, boolean) line: 145	
	org.apache.hadoop.yarn.util.ApplicationClassLoader.loadClass(java.lang.String) line: 126	
	javax.xml.parsers.FactoryFinder.getProviderClass(java.lang.String, java.lang.ClassLoader, boolean, boolean) line: 112	
	javax.xml.parsers.FactoryFinder.newInstance(java.lang.String, java.lang.ClassLoader, boolean, boolean) line: 170	
	javax.xml.parsers.FactoryFinder.newInstance(java.lang.String, java.lang.ClassLoader, boolean) line: 147	
	javax.xml.parsers.FactoryFinder.find(java.lang.String, java.lang.String) line: 257	
	javax.xml.parsers.DocumentBuilderFactory.newInstance() line: 121	
	org.apache.hadoop.mapred.JobConf(org.apache.hadoop.conf.Configuration).loadResource(java.util.Properties, org.apache.hadoop.conf.Configuration$Resource, boolean) line: 1884	
	org.apache.hadoop.mapred.JobConf(org.apache.hadoop.conf.Configuration).loadResources(java.util.Properties, java.util.ArrayList<org.apache.hadoop.conf.Configuration.Resource>, boolean) line: 1861	
	org.apache.hadoop.mapred.JobConf(org.apache.hadoop.conf.Configuration).getProps() line: 1778	
	org.apache.hadoop.mapred.JobConf(org.apache.hadoop.conf.Configuration).get(java.lang.String) line: 713	
	org.apache.hadoop.mapred.JobConf(org.apache.hadoop.conf.Configuration).getTrimmed(java.lang.String) line: 732	
	org.apache.hadoop.mapred.JobConf(org.apache.hadoop.conf.Configuration).getClass(java.lang.String, java.lang.Class<?>) line: 1574	
	org.apache.hadoop.fs.FileSystem.getFileSystemClass(java.lang.String, org.apache.hadoop.conf.Configuration) line: 2259	
	org.apache.hadoop.fs.FileSystem.createFileSystem(java.net.URI, org.apache.hadoop.conf.Configuration) line: 2272	
	org.apache.hadoop.fs.FileSystem.access$200(java.net.URI, org.apache.hadoop.conf.Configuration) line: 86	
	org.apache.hadoop.fs.FileSystem$Cache.getInternal(java.net.URI, org.apache.hadoop.conf.Configuration, org.apache.hadoop.fs.FileSystem$Cache$Key) line: 2311	
	org.apache.hadoop.fs.FileSystem$Cache.get(java.net.URI, org.apache.hadoop.conf.Configuration) line: 2293	
	org.apache.hadoop.fs.FileSystem.get(java.net.URI, org.apache.hadoop.conf.Configuration) line: 317	
	org.apache.hadoop.fs.FileSystem.getLocal(org.apache.hadoop.conf.Configuration) line: 288	
	org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(org.apache.hadoop.conf.Configuration) line: 270	
	org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(java.lang.String, org.apache.hadoop.conf.Configuration) line: 428	
	org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(java.lang.String, org.apache.hadoop.conf.Configuration) line: 164	
	org.apache.hadoop.mapred.YarnChild.configureLocalDirs(org.apache.hadoop.mapred.Task, org.apache.hadoop.mapred.JobConf) line: 225	
	org.apache.hadoop.mapred.YarnChild.configureTask(org.apache.hadoop.mapred.Task, org.apache.hadoop.security.Credentials, org.apache.hadoop.security.token.Token<org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier>) line: 277	
	org.apache.hadoop.mapred.YarnChild.main(java.lang.String[]) line: 142	
{noformat}
                
> application classloader may be used too early to load classes
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-5146
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5146
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task
>    Affects Versions: 2.0.3-alpha
>            Reporter: Sangjin Lee
>
> At least in the case of YarnChild, the application classloader is set fairly early (both in Configuration and as a TCCL). This has an effect of using the application classloader unexpectedly early.
> There is a fair amount of code that gets invoked between setting the classloader and executing mapper/reducer task.
> For example, I saw that the application classloader was asked to load a DOM parser class (com.sun.org.apache.xerces...) as part of initializing the filesystem. Luckily, in most cases this would be delegated to the parent classloader as the job classpath would not have those classes.
> However, in general, this behavior carries the risk of loading the same class twice, and potentially causing problems such as ClassCastException. Those would turn into nasty bugs that are hard to fix.
> It would be good to either set the application classloader as late as possible or place clearer limitations so it loads only the mapper/reducer classes and their dependencies.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira