You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Mitesh Singh Jat (JIRA)" <ji...@apache.org> on 2011/03/18 10:49:29 UTC
[jira] Created: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
NativeMapReduce does not Allow Configuration Parameters containing Spaces
--------------------------------------------------------------------------
Key: PIG-1917
URL: https://issues.apache.org/jira/browse/PIG-1917
Project: Pig
Issue Type: Bug
Reporter: Mitesh Singh Jat
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with ''wordcount'' as Native M-R job.
{code:title=nativeMR.pig}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
'' -Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{monospace}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{monospace}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{monospace}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{monospace}
Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Mitesh Singh Jat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mitesh Singh Jat updated PIG-1917:
----------------------------------
Description:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with _wordcount_ as Native M-R job.
{code:pig|title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{{
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
}}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{{
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
}}
Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
was:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with ''wordcount'' as Native M-R job.
{code:title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after ''-Dmapred.child.java.opts="-Xmx1536m'' makes
''-Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when ''-Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:pig|title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {{
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> }}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {{
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> }}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017197#comment-13017197 ]
Thejas M Nair commented on PIG-1917:
------------------------------------
The pig parser is parsing the argument "-Dmapred.child.java.opts='-Xmx1536m -Xss128k'" to native mapreduce as two arguments "-Dmapred.child.java.opts='-Xmx1536m" and "-Xss128k'" .
To support this use case, we need to parse it as one argument and remove the single quotes, similar to what bash shell does - "-Dmapred.child.java.opts=-Xmx1536m -Xss128k" (note the missing single quotes).
In general, quoted arguments should be unquoted before being given as argument to native-mapreduce or streaming command, to simulate what the shell does. This would be a change in behavior that is not backward compatible. But it is a problem only if users rely on the delimiter quotes being included in the argument.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Mitesh Singh Jat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mitesh Singh Jat updated PIG-1917:
----------------------------------
Description:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with _wordcount_ as Native M-R job.
{code:pig|title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
was:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with _wordcount_ as Native M-R job.
{code:pig|title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{{
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
}}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{{
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
}}
Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:pig|title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1917:
-------------------------------
Fix Version/s: (was: 0.8.0)
0.9.0
The change to parse "-Dmapred.child.java.opts='-Xmx1536m -Xss128k'" as one argument is also going to break backward compatibility, if any existing pig script is (accidentally) relying on that behavior. So making this change only for 0.9.
Also, instead of unquoting all arguments, only the "-D" property arguments will be unquoted. This change will only affect Native map-reduce statements (ie, not streaming command).
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017219#comment-13017219 ]
Thejas M Nair commented on PIG-1917:
------------------------------------
To avoid breaking backward compatibility while supporting this use case, one option would be to have a way to switch to the mode that supports this use case.
Maybe by adding a " using 'unquote-mode' " to the command ?
eg .
{code}
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int)
`wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir` using 'unquote-mode';
{code}
or maybe by setting a system property on commandline -
-Dpig.unquote.cmdstring='true'
I think adding a system property is better, as it does not pollute the syntax.
In 0.9, maybe we can make the unquote mode the default mode.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-1917:
--------------------------------
Fix Version/s: 0.8.0
Assignee: Thejas M Nair
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Mitesh Singh Jat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mitesh Singh Jat updated PIG-1917:
----------------------------------
Description:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with _wordcount_ as Native M-R job.
{code:title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
was:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with _wordcount_ as Native M-R job.
{code:pig|title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
_-Xss128k"_ as the first argument to _wordcount_
Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Mitesh Singh Jat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mitesh Singh Jat updated PIG-1917:
----------------------------------
Description:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with ''wordcount'' as Native M-R job.
{code:title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after ''-Dmapred.child.java.opts="-Xmx1536m'' makes
''-Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when ''-Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
was:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with ''wordcount'' as Native M-R job.
{code:title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
'' -Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with ''wordcount'' as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after ''-Dmapred.child.java.opts="-Xmx1536m'' makes
> ''-Xss128k"'' as the first argument to '''wordcount'''.
> Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when ''-Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019054#comment-13019054 ]
Richard Ding commented on PIG-1917:
-----------------------------------
+1
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-1917.1.patch
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Mitesh Singh Jat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mitesh Singh Jat updated PIG-1917:
----------------------------------
Description:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with ''wordcount'' as Native M-R job.
{code:title=nativeMR.pig|borderStyle=solid}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
'' -Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{code}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{code}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{code}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{code}
Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
was:
In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
Considering a simple pig script with ''wordcount'' as Native M-R job.
{code:title=nativeMR.pig}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}
The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
'' -Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{monospace}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{monospace}
On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{monospace}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...
{monospace}
Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with ''wordcount'' as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
> '' -Xss128k"'' as the first argument to '''wordcount'''.
> Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1917:
-------------------------------
Attachment: PIG-1917.1.patch
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-1917.1.patch
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-1917) NativeMapReduce does not Allow
Configuration Parameters containing Spaces
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair resolved PIG-1917.
--------------------------------
Resolution: Fixed
Unit test and test-patch passed.
Patch committed to trunk.
> NativeMapReduce does not Allow Configuration Parameters containing Spaces
> --------------------------------------------------------------------------
>
> Key: PIG-1917
> URL: https://issues.apache.org/jira/browse/PIG-1917
> Project: Pig
> Issue Type: Bug
> Reporter: Mitesh Singh Jat
> Assignee: Thejas M Nair
> Fix For: 0.9.0
>
> Attachments: PIG-1917.1.patch
>
>
> In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
> Considering a simple pig script with _wordcount_ as Native M-R job.
> {code:title=nativeMR.pig|borderStyle=solid}
> A = load 'input/WordCountInput.txt';
> B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
> store B into 'output/WordCount' using PigStorage;
> {code}
> The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
> _-Xss128k"_ as the first argument to _wordcount_
> Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
> Physical plan, Logical plan, and M-R plan of pig show double quotes.
> For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
> {code}
> MapReduce node scope-13
> hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
> --------
> {code}
> On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
> {code}
> $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
> 11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
> 11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
> 11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
> 11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
> 11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
> 11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
> ...
> ...
> {code}
> Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira