You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ahmed El Baz (JIRA)" <ji...@apache.org> on 2012/10/08 22:46:02 UTC
[jira] [Created] (HADOOP-8899) Classpath exceeds maximum OS limit
Ahmed El Baz created HADOOP-8899:
------------------------------------
Summary: Classpath exceeds maximum OS limit
Key: HADOOP-8899
URL: https://issues.apache.org/jira/browse/HADOOP-8899
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ahmed El Baz
We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472503#comment-13472503 ]
Alejandro Abdelnur commented on HADOOP-8899:
--------------------------------------------
You don't want to unjar and jar, as you'll lose in the process any JDK services definitions in the JARs. I'd suggest the following alternate approach: all JARs in the DistributedCache classpath should be symlink-ed to a distributedcache-lib/ dir in the current directory and then 'distributedcached-lib/*' should be added to the classpath.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Ahmed El Baz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed El Baz updated HADOOP-8899:
---------------------------------
Status: Patch Available (was: Open)
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478069#comment-13478069 ]
Suresh Srinivas commented on HADOOP-8899:
-----------------------------------------
Alejandro and Joep, please look at Ahmed's comments. I am +1 for this change given maven surefire.
I will commit this in a day or so, if there are no further comments. Comments post that can be addressed in a separate jira.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-8899:
------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Marking the jira as resolved as I had committed this patch sometime back. Thank you Ahmed.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Assignee: Ahmed El Baz
> Fix For: 1-win
>
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478665#comment-13478665 ]
Alejandro Abdelnur commented on HADOOP-8899:
--------------------------------------------
Suresh, my bad, I was missing the MANIFEST.MF *Class-path:* trick. Now it makes sense, no objection. I have not look at the patch but conceptually seems OK. Just make sure all paths to JARs listed in the *Class-path:* are relative to the location of the manifest only JAR and there are not ../ in the paths. In other words, all the JARs mentioned in the manifest only JAR must be in the same directory or in a subdirectory from where the manifest only JAR is. Thx
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472507#comment-13472507 ]
Alejandro Abdelnur commented on HADOOP-8899:
--------------------------------------------
The potential downside of my suggested approach is that you lose the capability to order JARs in the classpath as '*' will be resolved in system specific way (I've seen this being done differently in different unix versions); which in my opinion should not be an issue as you should not have duplicate JARs in your classpath to start. The other downside is that you won't, from the logs, know exactly what is in the classpath, you'll only see a 'distributedcache-lib/*' entry in the classpath.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Joep Rottinghuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472529#comment-13472529 ]
Joep Rottinghuis commented on HADOOP-8899:
------------------------------------------
Admittedly I have not kept up with this lately, but aren't symlinks (MKLINK) limited to NTFS?
Does that interopt well with Cygwin?
Not sure if that is truly a restriction anymore these days...
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478656#comment-13478656 ]
Suresh Srinivas commented on HADOOP-8899:
-----------------------------------------
Alejandro, this is manifest only jar. See similar things done in maven surefire - http://maven.apache.org/plugins/maven-surefire-plugin/examples/class-loading.html.
bq. then things will not work as expected as you'll lose some of those files in the consolidation
I do not understand given above comment. All you have done is create a jar with manifest file that has classpath. What do mean by things will not work as expected?
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472571#comment-13472571 ]
Alejandro Abdelnur commented on HADOOP-8899:
--------------------------------------------
Argh, forgot about that. Maybe in win distributed cache symlinks should be simulated with copies. Then the proposed approach would work
Alejandro
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas reassigned HADOOP-8899:
---------------------------------------
Assignee: Ahmed El Baz
Ahmed, I have added you a contributor to Hadoop Common. From now on you can assign the jiras to yourself.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Assignee: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478197#comment-13478197 ]
Alejandro Abdelnur commented on HADOOP-8899:
--------------------------------------------
I think the approach is not correct, reiterating on my prev comment. if more than one of the JARs being consolidated contains a META-INF service definition for the same service (ie o.a.h.fs.FileSystem, o.a.h.io.compress.CompressionCodec, o.a.h.security.SecurityInfo), then things will not work as expected as you'll lose some of those files in the consolidation.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Joep Rottinghuis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471855#comment-13471855 ]
Joep Rottinghuis commented on HADOOP-8899:
------------------------------------------
IIRC this is not an OS limit, but this is a limit of cmd.exe
Wouldn't jarring and unjarring end up being slow? (suppose better slow than not working).
You'd also have to make sure that the classpath order is preserved. Classloaders will generally load the first hit, so if you create your own jar you need to either drop duplicates, or make sure they are in order.
With signed jars duplicate entries can be a challenge.
I haven't looked at the code, but can this be avoided through the user of an environment variable instead of expanding the classpath on the command-line?
Along similar lines I recall switching from System.exec to the user of Procbuilder for similar problems. Not sure if relevant here.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Ahmed El Baz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed El Baz updated HADOOP-8899:
---------------------------------
Attachment: HADOOP-8899.branch-1-win.classpath.patch
Please note the following:
- The generated Jar does not encapsulate any other jars, it simply references all entries in the classpath in its Manifest, so this is just another way to propagate the long list of Jars to the tasks to launch.
- Classpath elements are referenced in the generated Jar's Manifest in the same order.
- This change is only gated on Windows where the Java classpath argument now has the generated Jar (which references the other ones), instead of the long list which is giving the problem. For non-Windows OS, the behavior is just the same as before.
- We do not explicitly unjar the file, we feed it to Jave to lookup the classpath elements from the manifest.
- The patch also includes testing performance changes (in TestJvmManager): in Windows, launchTask is only affected by ~3%, in Linux no impact since the logic is the same.
I have attached a patch with the changes (including test changes)
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8899) Classpath exceeds maximum OS limit
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472718#comment-13472718 ]
Hadoop QA commented on HADOOP-8899:
-----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548456/HADOOP-8899.branch-1-win.classpath.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1591//console
This message is automatically generated.
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira