You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2008/12/15 21:05:44 UTC
[jira] Created: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?
Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?
----------------------------------------------------------------------------------------------------------------------------
Key: PIG-564
URL: https://issues.apache.org/jira/browse/PIG-564
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Viraj Bhat
Fix For: types_branch
Consider the following Pig script which uses parameter substitution
{code}
%default qual '/user/viraj'
%default mydir 'mydir_myextraqual'
VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
dump VISIT_LOGS;
{code}
If you run the script as:
==================================================================================================================
java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
==================================================================================================================
You get the following error:
==================================================================================================================
2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:619)
java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
at org.apache.pig.PigServer.openIterator(PigServer.java:389)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
at org.apache.pig.Main.main(Main.java:306)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
... 6 more
==================================================================================================================
Also tried using: -param mydir='mydir\-myextraqual'
This behavior occurs if the parameter value contains characters such as +,=, ?.
A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
mydir='mydir-myextraqual' and then running the pig script as:
java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
The following issues need to be fixed:
1) In -param option if parameter value contains special characters, it is truncated
2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715767#action_12715767 ]
Olga Natkovich commented on PIG-564:
------------------------------------
Alan, thanks for review.
(1) output1.pig is a generated file. I think it was checked in initially by mistake. Its content is irrelevant.
(2) I might have to resubmit a patch anyway if I figure out the extra warnings (the link is broken at the moment). If I have to do that, I will also add comments.
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (PIG-564) Parameter Substitution using
-param option does not seem to work when parameters contain special
characters such as +,=,-,?,' "
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704455#action_12704455 ]
Viraj Bhat edited comment on PIG-564 at 4/29/09 8:04 PM:
---------------------------------------------------------
Another special character "/" is not handled correctly and is particularly useful when passing nested output directories.
It is truncated when it is passed to Pig.
Example case:
{code}
a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray);
b = foreach a generate id1;
store b into '/user/viraj/$outputfolder' using PigStorage();
{code}
Run this script as:
{code}
shell>hadoop fs -mkdir /user/viraj/paramtest
shell>java -cp pig.jar:/home/viraj/hadoop-0.18.0-dev/conf/ -Dhod.server='' org.apache.pig.Main -param outputfolder="paramtest/moretest" -r paramtest.pig
2009-04-30 03:00:21,234 [main] INFO org.apache.pig.Main - Dry run completed. Substituted pig script is at paramtest.pig.substituted
{code}
Now if we open the substituted Pig script (paramtest.pig.substituted).
{code}
a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray);
b = foreach a generate id1;
store b into '/user/viraj/paramtest' using PigStorage();
{code}
Viraj
was (Author: viraj):
Another special character "/" is not handled correctly and is particularly useful when passing nested output directories.
It is truncated when it is passed to Pig.
Example case:
{code}
a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray);
b = foreach a generate id1;
store b into '/user/viraj/$outputfolder' using PigStorage();
{code}
Run this script as:
{code}
shell>hadoop fs -mkdir /user/viraj/paramtest
shell>java -cp pig.jar:/home/viraj/hadoop-0.18.0-dev/conf/ -Dhod.server='' org.apache.pig.Main -param outputfolder="paramtest/moretest" paramtest.pig
2009-04-30 03:00:21,234 [main] INFO org.apache.pig.Main - Dry run completed. Substituted pig script is at paramtest.pig.substituted
{code}
Now if we open the substituted Pig script (paramtest.pig.substituted).
{code}
a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray);
b = foreach a generate id1;
store b into '/user/viraj/paramtest' using PigStorage();
{code}
Viraj
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?
Posted by "Romain Rigaux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12667988#action_12667988 ]
Romain Rigaux commented on PIG-564:
-----------------------------------
Another possible workaround seems to wrap the param between two ' or " and escape its beginning with a \
-param url='http://www.yahoo.com' --> http
-param url='\http://www.yahoo.com' --> http://www.yahoo.com
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Fix For: types_branch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715764#action_12715764 ]
Alan Gates commented on PIG-564:
--------------------------------
Questions/comments on the patch.
1) Why did output1.pig change to look exactly like the new input5.pig? It seems like output1.pig shouldn't have changed.
2) A comment in the javacc files on how OTHER and IDENTIFIER interact in the pattern matching might be helpful, as it isn't immediately obvious (at least to me :) ).
As long as 1 is ok, then +1.
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-564:
-------------------------------
Fix Version/s: 0.3.0
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Fix For: 0.3.0
>
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "David Ciemiewicz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696266#action_12696266 ]
David Ciemiewicz commented on PIG-564:
--------------------------------------
Period (.) is also a special character that seems to cause problems.
See related JIRA PIG-754
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715792#action_12715792 ]
Giridharan Kesavan commented on PIG-564:
----------------------------------------
Use this link for releaseaudit warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
I 've fixed the test-patch scripts for the broken link.
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-564:
-------------------------------
Status: Patch Available (was: Open)
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-564:
-------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
patch committed
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704455#action_12704455 ]
Viraj Bhat commented on PIG-564:
--------------------------------
Another special character "/" is not handled correctly and is particularly useful when passing nested output directories.
It is truncated when it is passed to Pig.
Example case:
{code}
a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray);
b = foreach a generate id1;
store b into '/user/viraj/$outputfolder' using PigStorage();
{code}
Run this script as:
{code}
shell>hadoop fs -mkdir /user/viraj/paramtest
shell>java -cp pig.jar:/home/viraj/hadoop-0.18.0-dev/conf/ -Dhod.server='' org.apache.pig.Main -param outputfolder="paramtest/moretest" paramtest.pig
2009-04-30 03:00:21,234 [main] INFO org.apache.pig.Main - Dry run completed. Substituted pig script is at paramtest.pig.substituted
{code}
Now if we open the substituted Pig script (paramtest.pig.substituted).
{code}
a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray);
b = foreach a generate id1;
store b into '/user/viraj/paramtest' using PigStorage();
{code}
Viraj
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-564:
---------------------------
Summary: Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' " (was: Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,? )
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Fix For: types_branch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-564:
----------------------------------
Assignee: Olga Natkovich
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715742#action_12715742 ]
Hadoop QA commented on PIG-564:
-------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409702/PIG-564.patch
against trunk revision 780722.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 159 release audit warnings (more than the trunk's current 156 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/console
This message is automatically generated.
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668183#action_12668183 ]
Viraj Bhat commented on PIG-564:
--------------------------------
There are certain cases where the param contains ' single quotes and '' double quotes.
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: types_branch
> Reporter: Viraj Bhat
> Fix For: types_branch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-564:
-------------------------------
Attachment: PIG-564.patch
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716246#action_12716246 ]
Hudson commented on PIG-564:
----------------------------
Integrated in Pig-trunk #463 (See [http://hudson.zones.apache.org/hudson/job/Pig-trunk/463/])
: problem with parameter substitution and special charachters (olgan)
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param
option does not seem to work when parameters contain special characters
such as +,=,-,?,' "
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715977#action_12715977 ]
Olga Natkovich commented on PIG-564:
------------------------------------
Thanks, Giri. I looked at warnings anf they are result of adding new test files for parameter substitution that don't have apache license because they don't support comments. So nothing to fix there.
I will be committing the patch later today.
> Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' "
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-564
> URL: https://issues.apache.org/jira/browse/PIG-564
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.2.0
> Reporter: Viraj Bhat
> Assignee: Olga Natkovich
> Attachments: PIG-564.patch
>
>
> Consider the following Pig script which uses parameter substitution
> {code}
> %default qual '/user/viraj'
> %default mydir 'mydir_myextraqual'
> VISIT_LOGS = load '$qual/$mydir' as (a,b,c);
> dump VISIT_LOGS;
> {code}
> If you run the script as:
> ==================================================================================================================
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig
> ==================================================================================================================
> You get the following error:
> ==================================================================================================================
> 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist
> at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59)
> at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED]
> at org.apache.pig.PigServer.openIterator(PigServer.java:389)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> ... 6 more
> ==================================================================================================================
> Also tried using: -param mydir='mydir\-myextraqual'
> This behavior occurs if the parameter value contains characters such as +,=, ?.
> A workaround for this behavior is using a param_file which contains <param_name>=<param_value> on each line, with the <param_value> enclosed by quotes. For example:
> mydir='mydir-myextraqual' and then running the pig script as:
> java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig
> The following issues need to be fixed:
> 1) In -param option if parameter value contains special characters, it is truncated
> 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
> 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.