You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ivan Mitic (JIRA)" <ji...@apache.org> on 2013/08/31 22:27:51 UTC

[jira] [Commented] (HADOOP-9924) FileUtil.createJarWithClassPath() does not generate relative classpath correctly

    [ https://issues.apache.org/jira/browse/HADOOP-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13755589#comment-13755589 ] 

Ivan Mitic commented on HADOOP-9924:
------------------------------------

Thanks Shanyu for reporting the problem and preparing the patch.

Fix looks good overall. Have just two minor comments:
1. I think you'll want to use Path#isAbsolute() to validate whether a path is relative or absolute on Windows. The problem with File#isAbsolute() is that it returns false for paths like "\\some\\path".
2. Nit:
{code}
+        File fileCpEntry = new File(classPathEntry);
+        if(!fileCpEntry.isAbsolute()) {
+          fileCpEntry = new File(workingDir, classPathEntry);
+        }
{code}
Can you factor the code such that File object is not created twice for relative paths?
                
> FileUtil.createJarWithClassPath() does not generate relative classpath correctly
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-9924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9924
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.1.0-beta, 0.23.9
>            Reporter: shanyu zhao
>            Assignee: shanyu zhao
>         Attachments: HADOOP-9924.patch
>
>
> On Windows, FileUtil.createJarWithClassPath() is called to generate a manifest jar file to pack classpath - to avoid the problem of classpath being too long.
> However, the relative classpath is not handled correctly. It relies on Java's File(relativePath) to resolve the relative path. But it really should be using the given pwd parameter to resolve the relative path.
> To reproduce this bug, you can try some pig job on Windows, it will fail and the pig log on the application master will look like this:
> 2013-08-29 23:25:55,498 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat not found
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat not found
> This is because the PigOutputFormat class is in the job.jar file but the classpath manifest has:
> file:/c:/apps/dist/hadoop-2.1.0-beta/bin/job.jar/job.jar
> When it really should be:
> file:/<job container folder>/job.jar/job.jar

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira