You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by rakesh kothari <rk...@hotmail.com> on 2010/10/07 20:46:38 UTC
Passing parameters to Pig Script using Java
Hi,
I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?
Thanks,
-Rakesh
Re: Passing parameters to Pig Script using Java
Posted by Julien Le Dem <le...@yahoo-inc.com>.
Oops, sorry
Yes this is old code.
Thanks for pointing this out.
This should give a hint though.
Julien
On 10/7/10 6:16 PM, "Jeff Zhang" <zj...@gmail.com> wrote:
Hi Julien,
You did what we have done in Pig 0.8. There's a little difference
between your api and ours. We encapsulates the parameter in Map rather
than List.
On Fri, Oct 8, 2010 at 4:43 AM, Julien Le Dem <le...@yahoo-inc.com> wrote:
> Here's my workaround:
> I extend PigServer with the following code copy/pasted from other places in Pig.
> Parameters are in the form: "foo=bar"
> Julien
>
> import org.apache.pig.ExecType;
> import org.apache.pig.backend.executionengine.ExecException;
> import org.apache.pig.backend.executionengine.ExecJob;
> import org.apache.pig.impl.PigContext;
> import org.apache.pig.tools.grunt.GruntParser;
> import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
>
>
> /**
> * adapted from org.apache.pig.Main
> * returns the stream of final pig script to be passed to Grunt
> */
> private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
> List<String> paramFiles, String scriptFile)
> throws org.apache.pig.tools.parameters.ParseException, IOException{
> ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
> String[] type1 = new String[1];
> String[] type2 = new String[1];
>
> StringWriter writer = new StringWriter();
> psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer, params.size() > 0 ? params.toArray(type1) : null,
> paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);
>
> return new BufferedReader(new StringReader(writer.toString()));
>
> }
>
> /**
> * adapted original code from file to apply the preprocessor
> * @param filePath relative path of the pig script
> * @param params parameters to be applied by the preprocessor
> * @throws IOException
> */
> public void registerScript(String filePath, List<String> params) throws IOException {
> try {
> InputStream stream = new FileInputStream(filePath);
> try {
> GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
> grunt.setInteractive(false);
> grunt.setParams(this);
> grunt.parseStopOnError(true);
> } finally {
> try {
> stream.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
> throw new IOException("Error while parsing script: "+filePath,e);
> } catch (org.apache.pig.tools.parameters.ParseException e) {
> throw new IOException("Error while parsing parameters: "+params,e);
> }
> }
>
> /**
> * adapted original code from file to apply the preprocessor
> * @param filePath relative path of the pig script
> * @param params parameters to be applied by the preprocessor
> * @throws IOException
> */
> public void registerScript(String filePath, String... params) throws IOException {
> registerScript(filePath, Arrays.asList(params));
> }
>
>
>
>
>
> On 10/7/10 1:27 PM, "Olga Natkovich" <ol...@yahoo-inc.com> wrote:
>
> Not at this point as parameter substitution is implemented a s preprocessor on the script.
>
> Olga
>
> -----Original Message-----
> From: rakesh kothari [mailto:rkothari_iit@hotmail.com]
> Sent: Thursday, October 07, 2010 11:47 AM
> To: pig-user@hadoop.apache.org
> Subject: Passing parameters to Pig Script using Java
>
>
> Hi,
>
> I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?
>
> Thanks,
> -Rakesh
>
>
>
--
Best Regards
Jeff Zhang
Re: Passing parameters to Pig Script using Java
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Julien,
You did what we have done in Pig 0.8. There's a little difference
between your api and ours. We encapsulates the parameter in Map rather
than List.
On Fri, Oct 8, 2010 at 4:43 AM, Julien Le Dem <le...@yahoo-inc.com> wrote:
> Here's my workaround:
> I extend PigServer with the following code copy/pasted from other places in Pig.
> Parameters are in the form: "foo=bar"
> Julien
>
> import org.apache.pig.ExecType;
> import org.apache.pig.backend.executionengine.ExecException;
> import org.apache.pig.backend.executionengine.ExecJob;
> import org.apache.pig.impl.PigContext;
> import org.apache.pig.tools.grunt.GruntParser;
> import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
>
>
> /**
> * adapted from org.apache.pig.Main
> * returns the stream of final pig script to be passed to Grunt
> */
> private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
> List<String> paramFiles, String scriptFile)
> throws org.apache.pig.tools.parameters.ParseException, IOException{
> ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
> String[] type1 = new String[1];
> String[] type2 = new String[1];
>
> StringWriter writer = new StringWriter();
> psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer, params.size() > 0 ? params.toArray(type1) : null,
> paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);
>
> return new BufferedReader(new StringReader(writer.toString()));
>
> }
>
> /**
> * adapted original code from file to apply the preprocessor
> * @param filePath relative path of the pig script
> * @param params parameters to be applied by the preprocessor
> * @throws IOException
> */
> public void registerScript(String filePath, List<String> params) throws IOException {
> try {
> InputStream stream = new FileInputStream(filePath);
> try {
> GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
> grunt.setInteractive(false);
> grunt.setParams(this);
> grunt.parseStopOnError(true);
> } finally {
> try {
> stream.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
> throw new IOException("Error while parsing script: "+filePath,e);
> } catch (org.apache.pig.tools.parameters.ParseException e) {
> throw new IOException("Error while parsing parameters: "+params,e);
> }
> }
>
> /**
> * adapted original code from file to apply the preprocessor
> * @param filePath relative path of the pig script
> * @param params parameters to be applied by the preprocessor
> * @throws IOException
> */
> public void registerScript(String filePath, String... params) throws IOException {
> registerScript(filePath, Arrays.asList(params));
> }
>
>
>
>
>
> On 10/7/10 1:27 PM, "Olga Natkovich" <ol...@yahoo-inc.com> wrote:
>
> Not at this point as parameter substitution is implemented a s preprocessor on the script.
>
> Olga
>
> -----Original Message-----
> From: rakesh kothari [mailto:rkothari_iit@hotmail.com]
> Sent: Thursday, October 07, 2010 11:47 AM
> To: pig-user@hadoop.apache.org
> Subject: Passing parameters to Pig Script using Java
>
>
> Hi,
>
> I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?
>
> Thanks,
> -Rakesh
>
>
>
--
Best Regards
Jeff Zhang
Re: Passing parameters to Pig Script using Java
Posted by Julien Le Dem <le...@yahoo-inc.com>.
Here's my workaround:
I extend PigServer with the following code copy/pasted from other places in Pig.
Parameters are in the form: "foo=bar"
Julien
import org.apache.pig.ExecType;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.backend.executionengine.ExecJob;
import org.apache.pig.impl.PigContext;
import org.apache.pig.tools.grunt.GruntParser;
import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;
/**
* adapted from org.apache.pig.Main
* returns the stream of final pig script to be passed to Grunt
*/
private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
List<String> paramFiles, String scriptFile)
throws org.apache.pig.tools.parameters.ParseException, IOException{
ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
String[] type1 = new String[1];
String[] type2 = new String[1];
StringWriter writer = new StringWriter();
psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer, params.size() > 0 ? params.toArray(type1) : null,
paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);
return new BufferedReader(new StringReader(writer.toString()));
}
/**
* adapted original code from file to apply the preprocessor
* @param filePath relative path of the pig script
* @param params parameters to be applied by the preprocessor
* @throws IOException
*/
public void registerScript(String filePath, List<String> params) throws IOException {
try {
InputStream stream = new FileInputStream(filePath);
try {
GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
grunt.setInteractive(false);
grunt.setParams(this);
grunt.parseStopOnError(true);
} finally {
try {
stream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
} catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
throw new IOException("Error while parsing script: "+filePath,e);
} catch (org.apache.pig.tools.parameters.ParseException e) {
throw new IOException("Error while parsing parameters: "+params,e);
}
}
/**
* adapted original code from file to apply the preprocessor
* @param filePath relative path of the pig script
* @param params parameters to be applied by the preprocessor
* @throws IOException
*/
public void registerScript(String filePath, String... params) throws IOException {
registerScript(filePath, Arrays.asList(params));
}
On 10/7/10 1:27 PM, "Olga Natkovich" <ol...@yahoo-inc.com> wrote:
Not at this point as parameter substitution is implemented a s preprocessor on the script.
Olga
-----Original Message-----
From: rakesh kothari [mailto:rkothari_iit@hotmail.com]
Sent: Thursday, October 07, 2010 11:47 AM
To: pig-user@hadoop.apache.org
Subject: Passing parameters to Pig Script using Java
Hi,
I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?
Thanks,
-Rakesh
RE: Passing parameters to Pig Script using Java
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
Not at this point as parameter substitution is implemented a s preprocessor on the script.
Olga
-----Original Message-----
From: rakesh kothari [mailto:rkothari_iit@hotmail.com]
Sent: Thursday, October 07, 2010 11:47 AM
To: pig-user@hadoop.apache.org
Subject: Passing parameters to Pig Script using Java
Hi,
I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?
Thanks,
-Rakesh