You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Joep Rottinghuis (JIRA)" <ji...@apache.org> on 2012/10/08 23:04:02 UTC

[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit

    [ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471855#comment-13471855 ] 

Joep Rottinghuis commented on HADOOP-8899:
------------------------------------------

IIRC this is not an OS limit, but this is a limit of cmd.exe

Wouldn't jarring and unjarring end up being slow? (suppose better slow than not working).
You'd also have to make sure that the classpath order is preserved. Classloaders will generally load the first hit, so if you create your own jar you need to either drop duplicates, or make sure they are in order.
With signed jars duplicate entries can be a challenge.

I haven't looked at the code, but can this be avoided through the user of an environment variable instead of expanding the classpath on the command-line?
Along similar lines I recall switching from System.exec to the user of Procbuilder for similar problems. Not sure if relevant here.
                
> Classpath exceeds maximum OS limit
> ----------------------------------
>
>                 Key: HADOOP-8899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 1-win
>            Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
>  java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira