You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "John Gordon (JIRA)" <ji...@apache.org> on 2012/06/26 01:55:43 UTC
[jira] [Created] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
John Gordon created MAPREDUCE-4368:
--------------------------------------
Summary: TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
Key: MAPREDUCE-4368
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 1-win
Environment: on Windows:
set PATH=%PATH%;"C:\this memorable place".
Reporter: John Gordon
Fix For: 1-win
TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401609#comment-13401609 ]
Hadoop QA commented on MAPREDUCE-4368:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533523/TaskRunner.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2517//console
This message is automatically generated.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: (was: TaskRunner.patch)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
Fixed patch apply issue in previous patch
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch, TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: (was: TaskRunner.patch)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
Patch for TaskRunner.java in 1-win
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401871#comment-13401871 ]
Bikas Saha commented on MAPREDUCE-4368:
---------------------------------------
In the latest patch, isnt the quote replacement too late? The initial split on ' ' would have already mangled the path that should have gone into javaOptsSplit[i].startsWith("-Djava.library.path="). So at this point javaOptsSplit[i].startsWith("-Djava.library.path=") contains an incomplete path.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401791#comment-13401791 ]
Hadoop QA commented on MAPREDUCE-4368:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533557/TaskRunner.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2518//console
This message is automatically generated.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner3.patch
Filed MAPREDUCE-4377 here (https://issues.apache.org/jira/browse/MAPREDUCE-4377) and added a comment for it.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch, TaskRunner2.patch, TaskRunner3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
Updated patch with other java.library.path case handled.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401819#comment-13401819 ]
Bikas Saha commented on MAPREDUCE-4368:
---------------------------------------
Thanks for the explanations. Scoping the quotes handling to Windows makes sense. In conjunction with the last comment on MAPREDUCE-4322 do you see some way by which we can reduce the number of Shell.WINDOWS in this code launch JVM code path?
To round off above examples, would you mind adding an example with only quotes and no spaces and explaining what happens?
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
Updated patch file with full paths, previous patch was taken from the mapred subdirectory.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Fix Version/s: (was: 1-win)
Labels: patch (was: )
Target Version/s: 1-win
Status: Patch Available (was: Open)
Remove embedded quotes specifically for java.library.path and refactor double for loop to reduce overhead.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: patch
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401177#comment-13401177 ]
Hadoop QA commented on MAPREDUCE-4368:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533422/TaskRunner.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2513//console
This message is automatically generated.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401863#comment-13401863 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
Good catch, Bikas. Thank you. Yes, this function in particular function would probably need a broader rewrite to handle all the escaping mechanisms and special characters on Linux. For instance, a string with -Djava.library.path=/home/jgordon/hadoop\ lib;/usr/lib64 would also have problems with surrounding quotes, because the \ would be interpreted literally.
touch this\ file
is different from:
touch "this\ file"
In that case, the more ideal solution would be to not pass javaopts as one string, but as an array of options or a hash a-priori to improve the ability to reason about them at all layers in the stack. Then we could have a check for non-escaped problem characters like ;|&\,whitespace,#,etc and quote escape the whole option only when we see those.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401889#comment-13401889 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
Actually, Bikas is right. We can't address user specified java.library.path -- only the java property we look up. The reason is that the user path is incomplete in the combination (quote + space) case. If we want to handle the user case, we have to either fix the parser or remove quotes across the board in the javaOptsSplit array, otherwise we will remove only the first quote and leave the second one dangling, which would leave a mismatched quote in the eventual command. Bringing the patch back to the system.getProperty scope.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401778#comment-13401778 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
Good catch on the break. I had trouble with the patches and manually generated one from the branch to address it in my last patch -- leaving the break in. That break is not supposed to be there. Resubmitting the patch.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Status: Open (was: Patch Available)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner2.patch
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch, TaskRunner2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401894#comment-13401894 ]
Bikas Saha commented on MAPREDUCE-4368:
---------------------------------------
That would leave the code change incomplete. But I am ok with that since it is blocked by the basic generic bug in the handling of the white space delimiter for java opts. Could you please open a jira (not targeted to branch-1-win) to track and fix the bug around handling blanks while delimiting java opts? And leave a comment with that jira number to explain the missing check mentioned in your last comment.
After that I am +1 on the fix. We can commit unless someone else has any objections.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch, TaskRunner2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-4368.
--------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
Looks good, +1.
I just committed this. Thanks for the patch John and Bikas/Ivan for reviews.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Assignee: John Gordon
> Labels: newbie, patch
> Fix For: 1-win
>
> Attachments: TaskRunner.patch, TaskRunner2.patch, TaskRunner3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401845#comment-13401845 ]
Bikas Saha commented on MAPREDUCE-4368:
---------------------------------------
Thanks. That makes sense. So for Windows we have made a solution for both 1) and 2) by taking advantage of the fact that " is an invalid path character and restricted it to java.library.path for safety.
The issue of spaces and quotes in paths remains open for Linux.
Wouldnt the fix also need to be applied to the path that comes with javaOptsSplit[i].startsWith("-Djava.library.path=")? Currently we are applying it to only libraryPath = System.getProperty("java.library.path")?
Would you also please mention this jira in your code comments so that people can refer back to it for a better understanding.
There is no need to "Submit Patch". It will always give an error because it tries to apply the patch to trunk and will fail. Just attach the file.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401104#comment-13401104 ]
Hadoop QA commented on MAPREDUCE-4368:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533410/TaskRunner.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2511//console
This message is automatically generated.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401898#comment-13401898 ]
Ivan Mitic commented on MAPREDUCE-4368:
---------------------------------------
Makes sense, +1, the latest patch looks good!
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch, TaskRunner2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401755#comment-13401755 ]
Bikas Saha commented on MAPREDUCE-4368:
---------------------------------------
1) I think spaces are allowed on Unix paths too. So this bug is not Windows specific. The fix need not be under Shell.WINDOWS. Also it does not seem specific to just the library path.
2) I think the refactoring ended up being incorrect because if(javaOptsSplit[i].startsWith("-Djava.library.path=")) then the new code will a) break and forget to add subsequent javaOptsSplit[i] and b) add only the system library path to vargs and forget to include the values in javaOptsSplit[i].
3) Also, the split on ' ' is done before appending the library path. So I fail to see how that affects spaces in the library path. It looks like when the library path with '"' is appended to "-Djava.library.path=" then the resulting string (when written to the batch script) does not end up being a valid command line argument. So the issue is not about splitting on the blank but about not escaping/handling the '"' already present in the path.
Would you mind spending some time and seeing if you can fix the generic problem? If its not worth a significant refactor then we could add some documentation around this instead of a fix.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: (was: TaskRunner.patch)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401887#comment-13401887 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
Good feedback, refactoring to keep the Shell.WINDOWS checks down and get the replace applied in both contexts iff Windows. Testing the new patch now, will post soon.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401116#comment-13401116 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
To explain why there are no tests added with this change -- existing commit tests find this issue before the patch on Windows if you add a quoted, space-separated, string to your PATH before running the tests.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401878#comment-13401878 ]
Ivan Mitic commented on MAPREDUCE-4368:
---------------------------------------
Thanks John for the patch, and Bikas for reviewing it thoroughly!
One minor comment on the latest patch, we would still need to scope the {{javaOptsSplit}} down to Windows only, and assign the value back. Instead of {code}javaOptsSplit[i].replaceAll("\"", "");{code} we should have
{code}if (Shell.WINDOWS) {
javaOptsSplit[i] = javaOptsSplit[i].replaceAll("\"", "");
}{code}
Otherwise, +1 from me!
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: (was: TaskRunner.patch)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-4368:
----------------------------------------
Assignee: John Gordon
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Assignee: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch, TaskRunner2.patch, TaskRunner3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Labels: newbie patch (was: patch)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401797#comment-13401797 ]
Bikas Saha commented on MAPREDUCE-4368:
---------------------------------------
b) add only the system library path to vargs and forget to include the values in javaOptsSplit[i].
This bug still seems to be there.
Did you get a chance to look at 3) above? Seems to me the issue is not about split on ' ' but not handling the '"' that gets embedded into the cmd line. You could add a random '"' in the path (and no space) and that should also repro the problem. Would you mind verifying that and seeing if its a Windows only issue? If its not, then it would be better to have generic solution.
The handling of spaces is itself an issue because the current code assumes there are no spaces except as delimiters. And that is not Windows specific.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401800#comment-13401800 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
for 1) We scoped it to Windows because in Windows " is not legal in a file name at all, but it is legal in filenames in Posix. While "file name" is similarly used in linux and windows, \"file\"\ name is also a legal construct in linux for a file that actually contains quotes in the path. In that case, stripping internal quotes would potentially break execution and the shell would have handled everything just fine. I did initially experiment with doing this for all arguments, in hopes of improving upstream quality for things like pig.additional.jars where we may find quotes in the path specified there. However, we kept it scoped down to java.library.path here because it is the only one explicitly interpreted as a path here. One can imagine a scenario where an upstream component has an argument like pig.query=^"a=load ...^" -- which may need embedded quotes and may also be escaped properly. In such a case, blanket removal of embedded quotes without handling escaping would be potentially dangerous.
3) is particularly interesting and maybe needs examples to explain.
Here is a javaopts string that the space split makes sense for:
-Dpig.additional.jars=/tmp/foo.jar -Djava.library.path=/usr/lib/java/;/usr/lib64/java -esa
This parses into this javaoptssplit:
-Dpig.additional.jars=/tmp/foo.jar
-Djava.library.path=/usr/lib/java;/usr/lib64/java
-esa
Which goes to the command line as:
java "-Dpig.additional.jars=/tmp/foo.jar" "-Djava.library.path=/usr/lib/java;/usr/lib64/java" "-esa" job.jar
Now, here is an example of what this patch fixes and why the combination of space splitting and quote escaping is broken:
-Dpig.additional.jars=/tmp/foo.jar -Djava.library.path=/usr/lib/java;/usr/lib64/java;"/opt/hadoop/somewhat memorable/path";/lib/java -esa
This parses into this javaoptssplit:
-Dpig.additional.jars=/tmp/foo.jar
-Djava.library.path=/usr/lib/java;/usr/lib64/java;"/opt/hadoop/somewhat
memorable/path";/lib/java
-esa
Which goes to the command line as:
java "-Dpig.additional.jars=/tmp/foo.jar" "-Djava.library.path=/usr/lib/java;/usr/lib64/java;"/opt/hadoop/somewhat" "memorable/path";/lib/java" "-esa" job.jar
This is bad in a number of ways, but here are the two most obvious to me:
1.) The semicolon is no longer quoted, so the shell read end of line after <path"> and will try to execute /lib/java as a separate shell command after java completes.
2.) java's command-line parsing rules will cause it to interpret "memorable/path" as the name of the jar it should execute and will yield (most likely) a ClassNoDefFoundException for "memorable/path" or "memorable.path".
Hope that explains what the comment was trying to read.
Thanks, Bikas, for the feedback -- really appreciate it. Great catch on the break!
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: (was: TaskRunner.patch)
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Gordon updated MAPREDUCE-4368:
-----------------------------------
Attachment: TaskRunner.patch
Adding JIRA # to the code comment for the fix.
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4368) TaskRunner fails to start jars
when the java.library.path contains a quoted path with embedded spaces
Posted by "John Gordon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401832#comment-13401832 ]
John Gordon commented on MAPREDUCE-4368:
----------------------------------------
On quotes with no spaces, here are two example input strings with an option that has quotes but no embedded spaces:
[EX 1 -- problem]
-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars";/lib/java -esa
This splits to:
-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars";/lib/java
-esa
then goes to the command line as:
java "-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars";/lib/java" "-esa" job.jar
This still has the semicolon termination problem due to the " being present, it is interpreted by the shell parser (on win/linux) as:
java "-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars"
<semicolon line break>
/lib/java" "-esa" job.jar
[EX 2 -- no problem]
-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars" -esa
This splits to:
-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars"
-esa
then goes to the command line as:
java "-Djava.library.path=/usr/lib/java/;/usr/lib64/java;"/opt/sun/sunstudio/somejars"" "-esa" job.jar
In this case, the quotes are interpreted by the command-line parser wrong, but in a benign way and it has no effect. The two problematic cases for this code around quotes are:
1.) "string with spaces"
2.) "anything" [ ; | > && || ] -- any quoted string followed by a shell character that is implicitly or explicitly end-of-command
> TaskRunner fails to start jars when the java.library.path contains a quoted path with embedded spaces
> -----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4368
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4368
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1-win
> Environment: on Windows:
> set PATH=%PATH%;"C:\this memorable place".
> Reporter: John Gordon
> Labels: newbie, patch
> Attachments: TaskRunner.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> TaskRunner splits arguments by space before it adds them back to the vargs list, so it loses all context of quote escaped strings with embedded spaces. This gets fixed up later by wrapping all arguments with " -- so you get something like java "-D<opt>=<value>". This is problematic for paths with embedded spaces, where we end up creating "-D<opt>=<first part" "last part>". To java, the jar being run is last part. So with the environment above, you will see "ClassNoDefFoundError: memorable" and the jar will fail to start. In this particular case, we know that java.libarary.path contains paths and the tests often use %PATH% to seed this, so the fix is to remove embedded quotes in listed path elements because we know the aggregate will be quoted when the JVM is started.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira