You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Siddhi Borkar <si...@persistent.co.in> on 2013/02/19 12:58:40 UTC
RE: reading input parameters in a pig script
Hi ,
I need to pass parameters dynamically to a pig script. Is there any way to read the parameters passed and their corresponding values without giving the parameter names in the pig script?
Thanks,
Siddhi
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: reading input parameters in a pig script
Posted by Jonathan Coveney <jc...@gmail.com>.
ah, ok, I see what you are doing. You could do it with some very hacky
native Pig, but what I would say it to do what Prashant mentioned and use
either Java or Python for control flow (we support both). So all of the
logic about what scripts to run etc would be done outside of Pig, and then
you build and submit the Pig job via that api.
2013/2/21 Siddhi Borkar <si...@persistent.co.in>
> Thanks Jonathan for your prompt reply,
> The parameters that will be supplied as the input to the pig script may or
> may not be present in the xml. For eg, say I pass fields as
> 'name,description,countDistributers' , the name and description can be
> obtained easily by parsing the xml , however for countDistributers' I have
> a separate pig script which needs to be invoked.
>
> I have read that it is not possible to have control statements (if else)
> in pig. Any idea how control flow can be defined?
> For eg:
> If fields contain countDistributers'
> Invoke countDistributers'.pig
>
> Also PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as
> (id:chararray, name:chararray, description:chararray)
>
> id:chararray, name:chararray, description:chararray has to be dynamically
> created based on the parameters passed.
> Is there any way of getting this done?
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:jcoveney@gmail.com]
> Sent: Wednesday, February 20, 2013 10:48 PM
> To: user@pig.apache.org
> Subject: Re: reading input parameters in a pig script
>
> what is going to be generating the "pig -param param1=..." and so on?
> Couldn't these be made into arguments? ie
>
> REGISTER /opt/apache_pig/pig-0.10.1/
> contrib/piggybank/java/piggybank.jar;
> REGISTER /tmp/custudf.jar;
>
> DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor('$fields');
> PRODUCTS = load 'product.xml' using
> org.apache.pig.piggybank.storage.XMLLoader('product') as (line:chararray);
> PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as
> (id:chararray, name:chararray, description:chararray);
>
>
> and you callit with pig -param fields=name,description
>
> and there has to be an output format, so in that case a %default would
> work?
>
>
> 2013/2/20 Siddhi Borkar <si...@persistent.co.in>
>
> > I will not be able to use %default statement in my pig script, as the
> > parameters being passed to my pig script are not fixed. I would need a
> > conditional check to be done in my pig script to check for each and
> > every input parameter if it is passed or not.
> > Also, there are no conditional operators (if/else) available in pig .
> >
> > Following is the psuedocode of the functionality I want to achieve
> >
> > Consider pig files:
> > 1) xmlparser.pig
> > 2) excelexporter.pig
> > 3) htmlexporter.pig
> >
> > 1) xmlparser.pig
> > REGISTER
> > /opt/apache_pig/pig-0.10.1/contrib/piggybank/java/piggybank.jar;
> > REGISTER /tmp/custudf.jar;
> >
> > DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor();
> > PRODUCTS = load 'product.xml' using
> > org.apache.pig.piggybank.storage.XMLLoader('product') as
> > (line:chararray); PRODUCT = FOREACH PRODUCTS GENERATE
> > FLATTEN(XMLProcessor(line)) as (id:chararray, name:chararray,
> > description:chararray);
> >
> > Please note, XMLProcessor is a custom java based udf which parses the
> xml.
> >
> > 2) excelexporter.pig
> > STORE PRODUCT INTO '/tmp/prod.csv' USING
> > CSVExcelStorage(',','NO_MULTILINE','UNIX');
> >
> > 3) htmlexporter.pig
> > //logic for this is not yet implemented
> >
> > Now the requirement is that I need to write a wrapper pig script which
> > invokes the following script and generates an output. The parameters
> > that will be passed are the input params and the out file format
> >
> > For ex pig -param param1=name param2=description outfileformat=csv
> > wrapper.pig
> >
> > Now what I need to do is based on the params passed to the wrapper pig
> > script, I need to send inputs to the xml parser and parse the input
> params.
> > In the above case since name and description are passed as params the
> > xml should be parsed only for these 2 fields.
> > Any idea how this can be achieved in a pig script?
> >
> > Also depending on the output file format, I need to invoke the
> > corresponding exporter script (html or csv) from my wrapper script. I
> > don’t see any conditional operators available (if/else) in pig. Any
> > idea how this can be achieved?
> >
> > -----Original Message-----
> > From: Jonathan Coveney [mailto:jcoveney@gmail.com]
> > Sent: Wednesday, February 20, 2013 2:38 PM
> > To: user@pig.apache.org
> > Subject: Re: reading input parameters in a pig script
> >
> > Reiterating Prashant's comments.
> >
> > In the script though you can have a %default statement which will
> > define the default value for a parameter, which can also be overriden.
> > My guess is this might let you do what you want?
> >
> >
> > 2013/2/20 Prashant Kommireddi <pr...@gmail.com>
> >
> > > Hi Siddhi,
> > >
> > > "Is there any way to access these params in the script without
> > > referring to the param name?" -- how would you associate a param
> > > value
> > to pig statement?
> > >
> > > I am guessing in this case your pig script is also dynamically
> generated?
> > > You could use PigServer API
> > > http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html
> > > to generate params in a Java program and embed them into a script.
> > >
> > > -Prashant
> > >
> > >
> > > On Tue, Feb 19, 2013 at 3:44 PM, Siddhi Borkar <
> > > siddhi_borkar@persistent.co.in> wrote:
> > >
> > > >
> > > > Consider the following command
> > > > pig -param param1=test param2=test1 param3=test2 myscript.pig
> > > >
> > > > In my case the parameters are dynamic, as in I could either pass
> > > > param1 only or I could pass all three params or some extra params.
> > > >
> > > > Since the parameters are dynamic, in my pig script I will not be
> > > > able to refrence the parameters as '$param1' . Is there any way to
> > > > access these params in the script without referring to the param
> name?
> > > >
> > > > ________________________________________
> > > > From: Jonathan Coveney [jcoveney@gmail.com]
> > > > Sent: Tuesday, February 19, 2013 6:42 PM
> > > > To: user@pig.apache.org
> > > > Subject: Re: reading input parameters in a pig script
> > > >
> > > > Can you give an example of what you'd like this to look like?
> > > >
> > > >
> > > > 2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> > > >
> > > > > Hi ,
> > > > >
> > > > > I need to pass parameters dynamically to a pig script. Is there
> > > > > any way
> > > > to
> > > > > read the parameters passed and their corresponding values
> > > > > without
> > > giving
> > > > > the parameter names in the pig script?
> > > > >
> > > > > Thanks,
> > > > > Siddhi
> > > > >
> > > > > DISCLAIMER
> > > > > ==========
> > > > > This e-mail may contain privileged and confidential information
> > > > > which
> > > is
> > > > > the property of Persistent Systems Ltd. It is intended only for
> > > > > the use
> > > > of
> > > > > the individual or entity to which it is addressed. If you are
> > > > > not the intended recipient, you are not authorized to read,
> > > > > retain, copy,
> > > print,
> > > > > distribute or use this message. If you have received this
> > > > > communication
> > > > in
> > > > > error, please notify the sender and delete all copies of this
> > message.
> > > > > Persistent Systems Ltd. does not accept any liability for virus
> > > infected
> > > > > mails.
> > > > >
> > > >
> > > > DISCLAIMER
> > > > ==========
> > > > This e-mail may contain privileged and confidential information
> > > > which is the property of Persistent Systems Ltd. It is intended
> > > > only for the use
> > > of
> > > > the individual or entity to which it is addressed. If you are not
> > > > the intended recipient, you are not authorized to read, retain,
> > > > copy, print, distribute or use this message. If you have received
> > > > this communication
> > > in
> > > > error, please notify the sender and delete all copies of this
> message.
> > > > Persistent Systems Ltd. does not accept any liability for virus
> > > > infected mails.
> > > >
> > >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which
> > is the property of Persistent Systems Ltd. It is intended only for the
> > use of the individual or entity to which it is addressed. If you are
> > not the intended recipient, you are not authorized to read, retain,
> > copy, print, distribute or use this message. If you have received this
> > communication in error, please notify the sender and delete all copies
> of this message.
> > Persistent Systems Ltd. does not accept any liability for virus
> > infected mails.
> >
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
RE: reading input parameters in a pig script
Posted by Siddhi Borkar <si...@persistent.co.in>.
Thanks Jonathan for your prompt reply,
The parameters that will be supplied as the input to the pig script may or may not be present in the xml. For eg, say I pass fields as 'name,description,countDistributers' , the name and description can be obtained easily by parsing the xml , however for countDistributers' I have a separate pig script which needs to be invoked.
I have read that it is not possible to have control statements (if else) in pig. Any idea how control flow can be defined?
For eg:
If fields contain countDistributers'
Invoke countDistributers'.pig
Also PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as (id:chararray, name:chararray, description:chararray)
id:chararray, name:chararray, description:chararray has to be dynamically created based on the parameters passed.
Is there any way of getting this done?
-----Original Message-----
From: Jonathan Coveney [mailto:jcoveney@gmail.com]
Sent: Wednesday, February 20, 2013 10:48 PM
To: user@pig.apache.org
Subject: Re: reading input parameters in a pig script
what is going to be generating the "pig -param param1=..." and so on?
Couldn't these be made into arguments? ie
REGISTER /opt/apache_pig/pig-0.10.1/
contrib/piggybank/java/piggybank.jar;
REGISTER /tmp/custudf.jar;
DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor('$fields');
PRODUCTS = load 'product.xml' using
org.apache.pig.piggybank.storage.XMLLoader('product') as (line:chararray); PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as (id:chararray, name:chararray, description:chararray);
and you callit with pig -param fields=name,description
and there has to be an output format, so in that case a %default would work?
2013/2/20 Siddhi Borkar <si...@persistent.co.in>
> I will not be able to use %default statement in my pig script, as the
> parameters being passed to my pig script are not fixed. I would need a
> conditional check to be done in my pig script to check for each and
> every input parameter if it is passed or not.
> Also, there are no conditional operators (if/else) available in pig .
>
> Following is the psuedocode of the functionality I want to achieve
>
> Consider pig files:
> 1) xmlparser.pig
> 2) excelexporter.pig
> 3) htmlexporter.pig
>
> 1) xmlparser.pig
> REGISTER
> /opt/apache_pig/pig-0.10.1/contrib/piggybank/java/piggybank.jar;
> REGISTER /tmp/custudf.jar;
>
> DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor();
> PRODUCTS = load 'product.xml' using
> org.apache.pig.piggybank.storage.XMLLoader('product') as
> (line:chararray); PRODUCT = FOREACH PRODUCTS GENERATE
> FLATTEN(XMLProcessor(line)) as (id:chararray, name:chararray,
> description:chararray);
>
> Please note, XMLProcessor is a custom java based udf which parses the xml.
>
> 2) excelexporter.pig
> STORE PRODUCT INTO '/tmp/prod.csv' USING
> CSVExcelStorage(',','NO_MULTILINE','UNIX');
>
> 3) htmlexporter.pig
> //logic for this is not yet implemented
>
> Now the requirement is that I need to write a wrapper pig script which
> invokes the following script and generates an output. The parameters
> that will be passed are the input params and the out file format
>
> For ex pig -param param1=name param2=description outfileformat=csv
> wrapper.pig
>
> Now what I need to do is based on the params passed to the wrapper pig
> script, I need to send inputs to the xml parser and parse the input params.
> In the above case since name and description are passed as params the
> xml should be parsed only for these 2 fields.
> Any idea how this can be achieved in a pig script?
>
> Also depending on the output file format, I need to invoke the
> corresponding exporter script (html or csv) from my wrapper script. I
> don’t see any conditional operators available (if/else) in pig. Any
> idea how this can be achieved?
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:jcoveney@gmail.com]
> Sent: Wednesday, February 20, 2013 2:38 PM
> To: user@pig.apache.org
> Subject: Re: reading input parameters in a pig script
>
> Reiterating Prashant's comments.
>
> In the script though you can have a %default statement which will
> define the default value for a parameter, which can also be overriden.
> My guess is this might let you do what you want?
>
>
> 2013/2/20 Prashant Kommireddi <pr...@gmail.com>
>
> > Hi Siddhi,
> >
> > "Is there any way to access these params in the script without
> > referring to the param name?" -- how would you associate a param
> > value
> to pig statement?
> >
> > I am guessing in this case your pig script is also dynamically generated?
> > You could use PigServer API
> > http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html
> > to generate params in a Java program and embed them into a script.
> >
> > -Prashant
> >
> >
> > On Tue, Feb 19, 2013 at 3:44 PM, Siddhi Borkar <
> > siddhi_borkar@persistent.co.in> wrote:
> >
> > >
> > > Consider the following command
> > > pig -param param1=test param2=test1 param3=test2 myscript.pig
> > >
> > > In my case the parameters are dynamic, as in I could either pass
> > > param1 only or I could pass all three params or some extra params.
> > >
> > > Since the parameters are dynamic, in my pig script I will not be
> > > able to refrence the parameters as '$param1' . Is there any way to
> > > access these params in the script without referring to the param name?
> > >
> > > ________________________________________
> > > From: Jonathan Coveney [jcoveney@gmail.com]
> > > Sent: Tuesday, February 19, 2013 6:42 PM
> > > To: user@pig.apache.org
> > > Subject: Re: reading input parameters in a pig script
> > >
> > > Can you give an example of what you'd like this to look like?
> > >
> > >
> > > 2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> > >
> > > > Hi ,
> > > >
> > > > I need to pass parameters dynamically to a pig script. Is there
> > > > any way
> > > to
> > > > read the parameters passed and their corresponding values
> > > > without
> > giving
> > > > the parameter names in the pig script?
> > > >
> > > > Thanks,
> > > > Siddhi
> > > >
> > > > DISCLAIMER
> > > > ==========
> > > > This e-mail may contain privileged and confidential information
> > > > which
> > is
> > > > the property of Persistent Systems Ltd. It is intended only for
> > > > the use
> > > of
> > > > the individual or entity to which it is addressed. If you are
> > > > not the intended recipient, you are not authorized to read,
> > > > retain, copy,
> > print,
> > > > distribute or use this message. If you have received this
> > > > communication
> > > in
> > > > error, please notify the sender and delete all copies of this
> message.
> > > > Persistent Systems Ltd. does not accept any liability for virus
> > infected
> > > > mails.
> > > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Persistent Systems Ltd. It is intended
> > > only for the use
> > of
> > > the individual or entity to which it is addressed. If you are not
> > > the intended recipient, you are not authorized to read, retain,
> > > copy, print, distribute or use this message. If you have received
> > > this communication
> > in
> > > error, please notify the sender and delete all copies of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus
> > > infected mails.
> > >
> >
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which
> is the property of Persistent Systems Ltd. It is intended only for the
> use of the individual or entity to which it is addressed. If you are
> not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus
> infected mails.
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: reading input parameters in a pig script
Posted by Jonathan Coveney <jc...@gmail.com>.
what is going to be generating the "pig -param param1=..." and so on?
Couldn't these be made into arguments? ie
REGISTER /opt/apache_pig/pig-0.10.1/
contrib/piggybank/java/piggybank.jar;
REGISTER /tmp/custudf.jar;
DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor('$fields');
PRODUCTS = load 'product.xml' using
org.apache.pig.piggybank.storage.XMLLoader('product') as (line:chararray);
PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as
(id:chararray, name:chararray, description:chararray);
and you callit with pig -param fields=name,description
and there has to be an output format, so in that case a %default would work?
2013/2/20 Siddhi Borkar <si...@persistent.co.in>
> I will not be able to use %default statement in my pig script, as the
> parameters being passed to my pig script are not fixed. I would need a
> conditional check to be done in my pig script to check for each and every
> input parameter if it is passed or not.
> Also, there are no conditional operators (if/else) available in pig .
>
> Following is the psuedocode of the functionality I want to achieve
>
> Consider pig files:
> 1) xmlparser.pig
> 2) excelexporter.pig
> 3) htmlexporter.pig
>
> 1) xmlparser.pig
> REGISTER /opt/apache_pig/pig-0.10.1/contrib/piggybank/java/piggybank.jar;
> REGISTER /tmp/custudf.jar;
>
> DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor();
> PRODUCTS = load 'product.xml' using
> org.apache.pig.piggybank.storage.XMLLoader('product') as (line:chararray);
> PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as
> (id:chararray, name:chararray, description:chararray);
>
> Please note, XMLProcessor is a custom java based udf which parses the xml.
>
> 2) excelexporter.pig
> STORE PRODUCT INTO '/tmp/prod.csv' USING
> CSVExcelStorage(',','NO_MULTILINE','UNIX');
>
> 3) htmlexporter.pig
> //logic for this is not yet implemented
>
> Now the requirement is that I need to write a wrapper pig script which
> invokes the following script and generates an output. The parameters that
> will be passed are the input params and the out file format
>
> For ex pig -param param1=name param2=description outfileformat=csv
> wrapper.pig
>
> Now what I need to do is based on the params passed to the wrapper pig
> script, I need to send inputs to the xml parser and parse the input params.
> In the above case since name and description are passed as params the xml
> should be parsed only for these 2 fields.
> Any idea how this can be achieved in a pig script?
>
> Also depending on the output file format, I need to invoke the
> corresponding exporter script (html or csv) from my wrapper script. I don’t
> see any conditional operators available (if/else) in pig. Any idea how this
> can be achieved?
>
> -----Original Message-----
> From: Jonathan Coveney [mailto:jcoveney@gmail.com]
> Sent: Wednesday, February 20, 2013 2:38 PM
> To: user@pig.apache.org
> Subject: Re: reading input parameters in a pig script
>
> Reiterating Prashant's comments.
>
> In the script though you can have a %default statement which will define
> the default value for a parameter, which can also be overriden. My guess is
> this might let you do what you want?
>
>
> 2013/2/20 Prashant Kommireddi <pr...@gmail.com>
>
> > Hi Siddhi,
> >
> > "Is there any way to access these params in the script without
> > referring to the param name?" -- how would you associate a param value
> to pig statement?
> >
> > I am guessing in this case your pig script is also dynamically generated?
> > You could use PigServer API
> > http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html
> > to generate params in a Java program and embed them into a script.
> >
> > -Prashant
> >
> >
> > On Tue, Feb 19, 2013 at 3:44 PM, Siddhi Borkar <
> > siddhi_borkar@persistent.co.in> wrote:
> >
> > >
> > > Consider the following command
> > > pig -param param1=test param2=test1 param3=test2 myscript.pig
> > >
> > > In my case the parameters are dynamic, as in I could either pass
> > > param1 only or I could pass all three params or some extra params.
> > >
> > > Since the parameters are dynamic, in my pig script I will not be
> > > able to refrence the parameters as '$param1' . Is there any way to
> > > access these params in the script without referring to the param name?
> > >
> > > ________________________________________
> > > From: Jonathan Coveney [jcoveney@gmail.com]
> > > Sent: Tuesday, February 19, 2013 6:42 PM
> > > To: user@pig.apache.org
> > > Subject: Re: reading input parameters in a pig script
> > >
> > > Can you give an example of what you'd like this to look like?
> > >
> > >
> > > 2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> > >
> > > > Hi ,
> > > >
> > > > I need to pass parameters dynamically to a pig script. Is there
> > > > any way
> > > to
> > > > read the parameters passed and their corresponding values without
> > giving
> > > > the parameter names in the pig script?
> > > >
> > > > Thanks,
> > > > Siddhi
> > > >
> > > > DISCLAIMER
> > > > ==========
> > > > This e-mail may contain privileged and confidential information
> > > > which
> > is
> > > > the property of Persistent Systems Ltd. It is intended only for
> > > > the use
> > > of
> > > > the individual or entity to which it is addressed. If you are not
> > > > the intended recipient, you are not authorized to read, retain,
> > > > copy,
> > print,
> > > > distribute or use this message. If you have received this
> > > > communication
> > > in
> > > > error, please notify the sender and delete all copies of this
> message.
> > > > Persistent Systems Ltd. does not accept any liability for virus
> > infected
> > > > mails.
> > > >
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which is the property of Persistent Systems Ltd. It is intended only
> > > for the use
> > of
> > > the individual or entity to which it is addressed. If you are not
> > > the intended recipient, you are not authorized to read, retain,
> > > copy, print, distribute or use this message. If you have received
> > > this communication
> > in
> > > error, please notify the sender and delete all copies of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus
> > > infected mails.
> > >
> >
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
RE: reading input parameters in a pig script
Posted by Siddhi Borkar <si...@persistent.co.in>.
I will not be able to use %default statement in my pig script, as the parameters being passed to my pig script are not fixed. I would need a conditional check to be done in my pig script to check for each and every input parameter if it is passed or not.
Also, there are no conditional operators (if/else) available in pig .
Following is the psuedocode of the functionality I want to achieve
Consider pig files:
1) xmlparser.pig
2) excelexporter.pig
3) htmlexporter.pig
1) xmlparser.pig
REGISTER /opt/apache_pig/pig-0.10.1/contrib/piggybank/java/piggybank.jar;
REGISTER /tmp/custudf.jar;
DEFINE XMLProcessor org.sdc.map.processor.XMLProcessor();
PRODUCTS = load 'product.xml' using org.apache.pig.piggybank.storage.XMLLoader('product') as (line:chararray);
PRODUCT = FOREACH PRODUCTS GENERATE FLATTEN(XMLProcessor(line)) as (id:chararray, name:chararray, description:chararray);
Please note, XMLProcessor is a custom java based udf which parses the xml.
2) excelexporter.pig
STORE PRODUCT INTO '/tmp/prod.csv' USING CSVExcelStorage(',','NO_MULTILINE','UNIX');
3) htmlexporter.pig
//logic for this is not yet implemented
Now the requirement is that I need to write a wrapper pig script which invokes the following script and generates an output. The parameters that will be passed are the input params and the out file format
For ex pig -param param1=name param2=description outfileformat=csv wrapper.pig
Now what I need to do is based on the params passed to the wrapper pig script, I need to send inputs to the xml parser and parse the input params. In the above case since name and description are passed as params the xml should be parsed only for these 2 fields.
Any idea how this can be achieved in a pig script?
Also depending on the output file format, I need to invoke the corresponding exporter script (html or csv) from my wrapper script. I don’t see any conditional operators available (if/else) in pig. Any idea how this can be achieved?
-----Original Message-----
From: Jonathan Coveney [mailto:jcoveney@gmail.com]
Sent: Wednesday, February 20, 2013 2:38 PM
To: user@pig.apache.org
Subject: Re: reading input parameters in a pig script
Reiterating Prashant's comments.
In the script though you can have a %default statement which will define the default value for a parameter, which can also be overriden. My guess is this might let you do what you want?
2013/2/20 Prashant Kommireddi <pr...@gmail.com>
> Hi Siddhi,
>
> "Is there any way to access these params in the script without
> referring to the param name?" -- how would you associate a param value to pig statement?
>
> I am guessing in this case your pig script is also dynamically generated?
> You could use PigServer API
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html
> to generate params in a Java program and embed them into a script.
>
> -Prashant
>
>
> On Tue, Feb 19, 2013 at 3:44 PM, Siddhi Borkar <
> siddhi_borkar@persistent.co.in> wrote:
>
> >
> > Consider the following command
> > pig -param param1=test param2=test1 param3=test2 myscript.pig
> >
> > In my case the parameters are dynamic, as in I could either pass
> > param1 only or I could pass all three params or some extra params.
> >
> > Since the parameters are dynamic, in my pig script I will not be
> > able to refrence the parameters as '$param1' . Is there any way to
> > access these params in the script without referring to the param name?
> >
> > ________________________________________
> > From: Jonathan Coveney [jcoveney@gmail.com]
> > Sent: Tuesday, February 19, 2013 6:42 PM
> > To: user@pig.apache.org
> > Subject: Re: reading input parameters in a pig script
> >
> > Can you give an example of what you'd like this to look like?
> >
> >
> > 2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> >
> > > Hi ,
> > >
> > > I need to pass parameters dynamically to a pig script. Is there
> > > any way
> > to
> > > read the parameters passed and their corresponding values without
> giving
> > > the parameter names in the pig script?
> > >
> > > Thanks,
> > > Siddhi
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information
> > > which
> is
> > > the property of Persistent Systems Ltd. It is intended only for
> > > the use
> > of
> > > the individual or entity to which it is addressed. If you are not
> > > the intended recipient, you are not authorized to read, retain,
> > > copy,
> print,
> > > distribute or use this message. If you have received this
> > > communication
> > in
> > > error, please notify the sender and delete all copies of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus
> infected
> > > mails.
> > >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information
> > which is the property of Persistent Systems Ltd. It is intended only
> > for the use
> of
> > the individual or entity to which it is addressed. If you are not
> > the intended recipient, you are not authorized to read, retain,
> > copy, print, distribute or use this message. If you have received
> > this communication
> in
> > error, please notify the sender and delete all copies of this message.
> > Persistent Systems Ltd. does not accept any liability for virus
> > infected mails.
> >
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: reading input parameters in a pig script
Posted by Jonathan Coveney <jc...@gmail.com>.
Reiterating Prashant's comments.
In the script though you can have a %default statement which will define
the default value for a parameter, which can also be overriden. My guess is
this might let you do what you want?
2013/2/20 Prashant Kommireddi <pr...@gmail.com>
> Hi Siddhi,
>
> "Is there any way to access these params in the script without referring to
> the param name?" -- how would you associate a param value to pig statement?
>
> I am guessing in this case your pig script is also dynamically generated?
> You could use PigServer API
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html to
> generate params in a Java program and embed them into a script.
>
> -Prashant
>
>
> On Tue, Feb 19, 2013 at 3:44 PM, Siddhi Borkar <
> siddhi_borkar@persistent.co.in> wrote:
>
> >
> > Consider the following command
> > pig -param param1=test param2=test1 param3=test2 myscript.pig
> >
> > In my case the parameters are dynamic, as in I could either pass param1
> > only or I could pass all three params or some extra params.
> >
> > Since the parameters are dynamic, in my pig script I will not be able to
> > refrence the parameters as '$param1' . Is there any way to access these
> > params in the script without referring to the param name?
> >
> > ________________________________________
> > From: Jonathan Coveney [jcoveney@gmail.com]
> > Sent: Tuesday, February 19, 2013 6:42 PM
> > To: user@pig.apache.org
> > Subject: Re: reading input parameters in a pig script
> >
> > Can you give an example of what you'd like this to look like?
> >
> >
> > 2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> >
> > > Hi ,
> > >
> > > I need to pass parameters dynamically to a pig script. Is there any way
> > to
> > > read the parameters passed and their corresponding values without
> giving
> > > the parameter names in the pig script?
> > >
> > > Thanks,
> > > Siddhi
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information which
> is
> > > the property of Persistent Systems Ltd. It is intended only for the use
> > of
> > > the individual or entity to which it is addressed. If you are not the
> > > intended recipient, you are not authorized to read, retain, copy,
> print,
> > > distribute or use this message. If you have received this communication
> > in
> > > error, please notify the sender and delete all copies of this message.
> > > Persistent Systems Ltd. does not accept any liability for virus
> infected
> > > mails.
> > >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is
> > the property of Persistent Systems Ltd. It is intended only for the use
> of
> > the individual or entity to which it is addressed. If you are not the
> > intended recipient, you are not authorized to read, retain, copy, print,
> > distribute or use this message. If you have received this communication
> in
> > error, please notify the sender and delete all copies of this message.
> > Persistent Systems Ltd. does not accept any liability for virus infected
> > mails.
> >
>
Re: reading input parameters in a pig script
Posted by Prashant Kommireddi <pr...@gmail.com>.
Hi Siddhi,
"Is there any way to access these params in the script without referring to
the param name?" -- how would you associate a param value to pig statement?
I am guessing in this case your pig script is also dynamically generated?
You could use PigServer API
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html to
generate params in a Java program and embed them into a script.
-Prashant
On Tue, Feb 19, 2013 at 3:44 PM, Siddhi Borkar <
siddhi_borkar@persistent.co.in> wrote:
>
> Consider the following command
> pig -param param1=test param2=test1 param3=test2 myscript.pig
>
> In my case the parameters are dynamic, as in I could either pass param1
> only or I could pass all three params or some extra params.
>
> Since the parameters are dynamic, in my pig script I will not be able to
> refrence the parameters as '$param1' . Is there any way to access these
> params in the script without referring to the param name?
>
> ________________________________________
> From: Jonathan Coveney [jcoveney@gmail.com]
> Sent: Tuesday, February 19, 2013 6:42 PM
> To: user@pig.apache.org
> Subject: Re: reading input parameters in a pig script
>
> Can you give an example of what you'd like this to look like?
>
>
> 2013/2/19 Siddhi Borkar <si...@persistent.co.in>
>
> > Hi ,
> >
> > I need to pass parameters dynamically to a pig script. Is there any way
> to
> > read the parameters passed and their corresponding values without giving
> > the parameter names in the pig script?
> >
> > Thanks,
> > Siddhi
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is
> > the property of Persistent Systems Ltd. It is intended only for the use
> of
> > the individual or entity to which it is addressed. If you are not the
> > intended recipient, you are not authorized to read, retain, copy, print,
> > distribute or use this message. If you have received this communication
> in
> > error, please notify the sender and delete all copies of this message.
> > Persistent Systems Ltd. does not accept any liability for virus infected
> > mails.
> >
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
RE: reading input parameters in a pig script
Posted by Siddhi Borkar <si...@persistent.co.in>.
Consider the following command
pig -param param1=test param2=test1 param3=test2 myscript.pig
In my case the parameters are dynamic, as in I could either pass param1 only or I could pass all three params or some extra params.
Since the parameters are dynamic, in my pig script I will not be able to refrence the parameters as '$param1' . Is there any way to access these params in the script without referring to the param name?
________________________________________
From: Jonathan Coveney [jcoveney@gmail.com]
Sent: Tuesday, February 19, 2013 6:42 PM
To: user@pig.apache.org
Subject: Re: reading input parameters in a pig script
Can you give an example of what you'd like this to look like?
2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> Hi ,
>
> I need to pass parameters dynamically to a pig script. Is there any way to
> read the parameters passed and their corresponding values without giving
> the parameter names in the pig script?
>
> Thanks,
> Siddhi
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: reading input parameters in a pig script
Posted by Jonathan Coveney <jc...@gmail.com>.
Can you give an example of what you'd like this to look like?
2013/2/19 Siddhi Borkar <si...@persistent.co.in>
> Hi ,
>
> I need to pass parameters dynamically to a pig script. Is there any way to
> read the parameters passed and their corresponding values without giving
> the parameter names in the pig script?
>
> Thanks,
> Siddhi
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>