You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Yuri Pradkin (JIRA)" <ji...@apache.org> on 2009/12/09 19:52:18 UTC
[jira] Created: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
--------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-1286
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.20.2
Environment: Linux F12; streaming
Reporter: Yuri Pradkin
Assignee: Yuri Pradkin
I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Christian Kunz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Kunz reopened MAPREDUCE-1286:
---------------------------------------
I would vote to declare this as blocker for 0.20.
It not only affects hadoop streaming, but also system calls using the hadoop command line utility will fail like:
Exception in thread "main" java.lang.NoClassDefFoundError: "-Dhadoop/tasklog/taskid=attempt_201006181908_0498_m_000000_0
Caused by: java.lang.ClassNotFoundException: "-Dhadoop.tasklog.taskid=attempt_201006181908_0498_m_000000_0
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: "-Dhadoop.tasklog.taskid=attempt_201006181908_0498_m_000000_0. Program will exit.
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796744#action_12796744 ]
Hudson commented on MAPREDUCE-1286:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #196 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/])
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788338#action_12788338 ]
Hadoop QA commented on MAPREDUCE-1286:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427511/mr-1286
against trunk revision 888761.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/311/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/311/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/311/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/311/console
This message is automatically generated.
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Yuri Pradkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuri Pradkin updated MAPREDUCE-1286:
------------------------------------
Status: Patch Available (was: In Progress)
just remove the quotes
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Yuri Pradkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuri Pradkin updated MAPREDUCE-1286:
------------------------------------
Attachment: mr-1286
just remove quotes
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1286:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
The double quotes were added intentionally in MAPREDUCE-1086, but removing them looks OK.
+1 I committed this. Thanks, Yuri!
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Yuri Pradkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-1286 started by Yuri Pradkin.
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1286) Quotes in environment
HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with
something else
Posted by "Yuri Pradkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791598#action_12791598 ]
Yuri Pradkin commented on MAPREDUCE-1286:
-----------------------------------------
OK, can somebody commit this please? It's kind of trivial, but all the more reasons to close it.
> Quotes in environment HADOOP_CLIENT_OPTS confuse parsing if this env is concatenated with something else
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1286
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.2
> Environment: Linux F12; streaming
> Reporter: Yuri Pradkin
> Assignee: Yuri Pradkin
> Attachments: mr-1286
>
>
> I use streaming and in the perl-reducer I write to hdfs using a pipe to hdfs -put - .... It turns out that because TaskRunner sets the environment HADOOP_CLIENT_OPTS in double quotes, when hdfs shell script concatenates these with something else, the command fails: .e.g java -Dblah=x -Dfoo=y "-Dhadoop.tasklog.taskid=z -Dhadoop.tasklog.totalLogFileSize=s"...
> Since I don't see any reason to have these double quotes in the original code; I propose they're removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.