You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Grant Ingersoll (JIRA)" <ji...@apache.org> on 2011/06/14 02:26:47 UTC
[jira] [Created] (PIG-2122) Parameter Substitution doesn't work in
the Grunt shell
Parameter Substitution doesn't work in the Grunt shell
------------------------------------------------------
Key: PIG-2122
URL: https://issues.apache.org/jira/browse/PIG-2122
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.1, 0.8.0
Reporter: Grant Ingersoll
Priority: Minor
Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
#Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
{quote}
foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
Y = foreach foo generate *, '$time';
dump Y;
{quote}
Output:
{quote}
2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1 2 3,,,$time)
(4 5 6,,,$time)
{quote}
Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
{quote}
2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(1 2 3,,,FOO)
(4 5 6,,,FOO)
{quote}
Also, things like don't work (nor does %declare):
{quote}
grunt> %default DATE '20090101';
2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
Was expecting one of:
<EOF>
"cat" ...
"fs" ...
"sh" ...
"cd" ...
"cp" ...
"copyFromLocal" ...
"copyToLocal" ...
"dump" ...
"describe" ...
"aliases" ...
"explain" ...
"help" ...
"kill" ...
"ls" ...
"mv" ...
"mkdir" ...
"pwd" ...
"quit" ...
"register" ...
"rm" ...
"rmf" ...
"set" ...
"illustrate" ...
"run" ...
"exec" ...
"scriptDone" ...
"" ...
<EOL> ...
";" ...
Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Janne Jalkanen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259606#comment-13259606 ]
Janne Jalkanen commented on PIG-2122:
-------------------------------------
Upvoting; this makes debugging Pig scripts *really* hard using Grunt. I mostly run a part of the script simply by cut-n-pasting from the editor into a Grunt shell, then stop to examine the intermediate values. Having to do manual search-replace for variables makes them useless.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049910#comment-13049910 ]
Dmitriy V. Ryaboy commented on PIG-2122:
----------------------------------------
This really does diminish usefulness of Grunt significantly.
There isn't an easy way to apply the existing preprocessor here, which is a shame, but it's an issue that comes up often. It would be great if someone took a crack at fixing this.. theoretically having a in-memory variable map and doing line-by-line substitution before handing off to the parser should work, right?
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049922#comment-13049922 ]
Grant Ingersoll commented on PIG-2122:
--------------------------------------
I know it would be nice, but is back compat really a requirement for a pre 1.0 release?
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049926#comment-13049926 ]
Olga Natkovich commented on PIG-2122:
-------------------------------------
Backward compatibility is a requirement because we have many customers and many of them uset this feature especially in production environment.
In my opinion design/implementation that does not provide backward comptaibility for batch processing is a non-starter.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Joshua Hartman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464326#comment-13464326 ]
Joshua Hartman commented on PIG-2122:
-------------------------------------
This irritates me on a weekly basis. I'm often running ad-hoc analysis in grunt with snippets taken from some heavier scripts I've written and I get bitten ALL the time.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049847#comment-13049847 ]
Grant Ingersoll commented on PIG-2122:
--------------------------------------
Seems to me one of the major use cases of the Grunt shell is to test your scripts line by line. If you don't have parameter substitution, then you have to futz around with hand editing out the substitutions to run it in Grunt, and then add it back in before you submit it to run in file mode.
The other use case is, the documentation doesn't work! I would bet that many people simply try out what is on the website as examples, so when things like declare, etc. as copy and pasted from the website don't work, they lose confidence in the tool.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Gianmarco De Francisci Morales (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259783#comment-13259783 ]
Gianmarco De Francisci Morales commented on PIG-2122:
-----------------------------------------------------
On the backward compatibility issue, I must say that I think modifying Grunt and interactive mode is not as bad as modifying the batch mode.
I don't feel changing Grunt should be a big issue as most of the production job that I know are batch scripts which would be unaffected.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Jeff Wartes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101302#comment-13101302 ]
Jeff Wartes commented on PIG-2122:
----------------------------------
I found this JIRA issue because:
1) I was copying a script into the shell for testing and was confused why it was wasn't working.
2) Reading the documentation and attempting alternative methods to set parameters didn't work either.
I mention this because I hit *both* of cases previously presented as reasons to fix this issue. The second, because of the first.
If this issue cannot be fixed, I'd suggest the grunt shell throw a more helpful error when it encounters preprocessor directives, and warn when it sees likely attempts at dollar-sign interpolation.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "SnowCrash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470566#comment-13470566 ]
SnowCrash commented on PIG-2122:
--------------------------------
[~aniket486]
Your workaround is not really much help if you're trying to debug a script (i.e. line by line, DUMPing or ILLUSTRATing as you go. We want to be able to use the interactive shell with the existing script (including params) but execute it line by line.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049829#comment-13049829 ]
Olga Natkovich commented on PIG-2122:
-------------------------------------
This is by design. We did not see any need for parameter substitution in the interactive mode. The current implementation is based on a preprocessor applied to the entire script.
We would need a compelling case to reconsider.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Aniket Mokashi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466619#comment-13466619 ]
Aniket Mokashi commented on PIG-2122:
-------------------------------------
There is a workaround to this that you can use-
* Store your script line in a file (with $params included).
* Start grunt interactively
* type- run -param a=b -param c=d myscript.pig
This should do the trick.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049920#comment-13049920 ]
Olga Natkovich commented on PIG-2122:
-------------------------------------
I think the trick would be to maintain backward compatibility.
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2122) Parameter Substitution doesn't work
in the Grunt shell
Posted by "Gianmarco De Francisci Morales (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259781#comment-13259781 ]
Gianmarco De Francisci Morales commented on PIG-2122:
-----------------------------------------------------
Hi,
I can agree that the feature is useful.
However to debug a Pig script you can use the -debug and -dryrun options:
{code}
–dryrun
Flag. With this option, the script is not run and a fully
substituted Pig script produced in the current working
directory named original_script_name.substituted
{code}
This should help you avoid manual search/replace.
Cheers,
> Parameter Substitution doesn't work in the Grunt shell
> ------------------------------------------------------
>
> Key: PIG-2122
> URL: https://issues.apache.org/jira/browse/PIG-2122
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.8.1
> Reporter: Grant Ingersoll
> Priority: Minor
>
> Simple param substitution and things like %declare (as copied out of the docs) don't work in the grunt shell.
> #Start Pig with: Start Pig with: bin/pig -x local -p time=FOO
> {quote}
> foo = LOAD '/user/grant/foo.txt' AS (a:chararray, b:chararray, c:chararray);
> Y = foreach foo generate *, '$time';
> dump Y;
> {quote}
> Output:
> {quote}
> 2011-06-13 20:22:24,197 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,$time)
> (4 5 6,,,$time)
> {quote}
> Same script, stored in junk.pig, run as: bin/pig -x local -p time=FOO junk.pig
> {quote}
> 2011-06-13 20:23:38,864 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> (1 2 3,,,FOO)
> (4 5 6,,,FOO)
> {quote}
> Also, things like don't work (nor does %declare):
> {quote}
> grunt> %default DATE '20090101';
> 2011-06-13 20:18:19,943 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "%default "" at line 1, column 1.
> Was expecting one of:
> <EOF>
> "cat" ...
> "fs" ...
> "sh" ...
> "cd" ...
> "cp" ...
> "copyFromLocal" ...
> "copyToLocal" ...
> "dump" ...
> "describe" ...
> "aliases" ...
> "explain" ...
> "help" ...
> "kill" ...
> "ls" ...
> "mv" ...
> "mkdir" ...
> "pwd" ...
> "quit" ...
> "register" ...
> "rm" ...
> "rmf" ...
> "set" ...
> "illustrate" ...
> "run" ...
> "exec" ...
> "scriptDone" ...
> "" ...
> <EOL> ...
> ";" ...
>
> Details at logfile: /Users/grant.ingersoll/projects/apache/pig/release-0.8.1/pig_1308002917912.log
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira