You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Craig Macdonald <cr...@dcs.gla.ac.uk> on 2008/02/08 13:57:42 UTC
yahoo-specific pig - improvements to syntax and .pigrc
Good morning Pig-devs,
This email notes some of the yahoo specifics remaining in Pig that may
be needed to checked before a Pig release (see 1. below). I would hope
that Pig syntax can be evolved to allow these to be removed (see (2)
below), and instead placed in users .pigrc.
From PigContext, I note that the .pigrc is in fact a place for
properties. An alternative would be for the Grunt set command to set
System properties, and then make .pigrc into a pig script, allowing
users to define aliases, register common jar files, import common
namespaces, include other pig script files.
Please direct how JIRAs should be created to track these issues - one
issue for all, with subtasks; separate tasks for the three issues below?
Details below.
Craig
1. Yahoo specifics
src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
doHod() contains yahoo specific stuff. I'm not sure Hod has
stabalised sufficiently for this to be changed
fixUpDomain() assumes unqualified hostnames are part of the
.inktomisearch.com DNS domain
src/org/apache/pig/impl/PigContext.java:125
packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
Note - these is no Pig command to allow imports of package
namespaces into the packageImportList ArrayList
scripts/pig.pl
kryptontite mentions, specifics: 69, 114
2. Extensions to Pig syntax
(a) "set" command sets all system properties
(b) "include" includes and parses another pig script
(c) "import" adds a package namespace to the search path
3. Change so that ~/.pigrc into a pig script that is parsed on startup
of Grunt/PigServer?
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Stefan Groschupf <sg...@101tec.com>.
> I'm not a project lead but as a user of pig I'd like a possibility
> to configure pig programmatically.
+1
yea we really would love to have this too and most important have
access to this object within the load and store functions.
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Benjamin Francisoud <be...@joost.com>.
Craig Macdonald a écrit :
> PigContext can indeed wrap up existing Pig specific system properties,
> given the addition of a getProperty() method as well.
>
> For instance, many System properties are used to control HOD
> connection in HExecutionEngine. These could instead be stored in
> PigContext, which is passed to HExecutionEngine anyway.
+1
>
> Any thoughts about set, import, include etc?
>
> Using pigrc as a place for set commands etc would allow users to
> simply call pig.pl to start their Pig session - the pigrc could do
> imports, set appropriate properties etc. The set command may be tricky
> - if properties are used to control the connection to the execution
> engine, then changing these properties using a set command would have
> no effect, as PigContext.connect() has already been called by
> PigServer's constructor. One option might be for PigServer to delay
> connection until the first query is executed.
>
> C
Yes, there seems to be an issue with the current implementation and
PigContext/PigServer life cycle (see
https://issues.apache.org/jira/browse/PIG-93)
Avoiding har code configration using system properties in PigServer or
PigContext would allow to set the values using different approach
depending on the context.
Extracting the part (populateProps in my example bellow) from PigContext
and PigServer would allow other people (like me) to populate PigContext
differently.
For example reading xml custom configuration file, getting properties
from a spring bean or get properties from a web service.
I may be missing some "big picture" here since I don't know all pig code
base but on an architecture point of view it seems a better option to me...
+------------------+ +------------------+ +------------------+
| SystemPropConf | | PigContext | | PigServer |
|+----------------+| |+----------------+| |+----------------+|
|populateProps() | |setProperties() | |setPigContext() |
| |--->|getProperties() |--->|setPigContext() |
|+----------------+| |+----------------+| |connect() |
| | | | |registerQuery() |
| | | | |+----------------+|
+------------------+ +------------------+ +------------------+
Code would look something like:
PigContext context = new PigContext(execType);
context.setProperties(SystemPropConf.populateProps());
PigServer server = new PigServer();
server.setPigContext(context); // (or the actual contructor)
server.registerQuery()
server.store();
ps: doing "kind of" UML /ASCII art in mail is not easy :D
--
Benjamin Francisoud
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Craig Macdonald <cr...@dcs.gla.ac.uk>.
PigContext can indeed wrap up existing Pig specific system properties,
given the addition of a getProperty() method as well.
For instance, many System properties are used to control HOD connection
in HExecutionEngine. These could instead be stored in PigContext, which
is passed to HExecutionEngine anyway.
Any thoughts about set, import, include etc?
Using pigrc as a place for set commands etc would allow users to simply
call pig.pl to start their Pig session - the pigrc could do imports, set
appropriate properties etc. The set command may be tricky - if
properties are used to control the connection to the execution engine,
then changing these properties using a set command would have no effect,
as PigContext.connect() has already been called by PigServer's
constructor. One option might be for PigServer to delay connection until
the first query is executed.
C
Alan Gates wrote:
> It is already possible to do the following:
>
> PigContext context = new PigContext(execType); // execType is
> PigServer.ExecType
> PigServer pig = new PigServer(context);
> pig.registerQuery(...);
> ...
>
> PigContext contains, among other things, a Properties object that keep
> the properties. What's missing is a way to set values in that
> properties object. So if we added a setProperty() method to
> PigContext I think we'd have what you suggest below.
>
> Alan.
>
> Benjamin Francisoud wrote:
>> I'm not a project lead but as a user of pig I'd like a possibility to
>> configure pig programmatically.
>>
>> Doing something like:
>>
>> PigConfiguration conf = new PigConfiguration();
>> conf.set("foo", "bar");
>> Pig pig = new Pig();
>> pig.setConf(conf);
>> pig.run();
>>
>> This is just a pseudo-code example.
>> Replace Pig and PigConfiguration with whatever name you like ;)
>>
>> --
>> Benjamin Francisoud
>>
>> Craig Macdonald a écrit :
>>> Hi pig-devs,
>>>
>>> Just a quiet ping for comments from project leads on this. It seems
>>> I have raised several issues recently that require configuration of
>>> pig. Questions:
>>> * Are System properties the best place for these?
>>> * Should .pigrc evolve into a place for Pig aliases and properties,
>>> and even scripts? (similar to .bashrc etc)
>>> * Should new commands be added: import, include, sharedFS etc?
>>> * Please direct me as to how JIRAs should be created.
>>> I may be able to provide patches to some JIRAs I have created if we
>>> have a policy for configuration-type stuff.
>>>
>>> C
>>>
>>> Craig Macdonald wrote:
>>>> Good morning Pig-devs,
>>>>
>>>> This email notes some of the yahoo specifics remaining in Pig that
>>>> may be needed to checked before a Pig release (see 1. below). I
>>>> would hope that Pig syntax can be evolved to allow these to be
>>>> removed (see (2) below), and instead placed in users .pigrc.
>>>>
>>>> From PigContext, I note that the .pigrc is in fact a place for
>>>> properties. An alternative would be for the Grunt set command to
>>>> set System properties, and then make .pigrc into a pig script,
>>>> allowing users to define aliases, register common jar files, import
>>>> common namespaces, include other pig script files.
>>>>
>>>> Please direct how JIRAs should be created to track these issues -
>>>> one issue for all, with subtasks; separate tasks for the three
>>>> issues below?
>>>>
>>>> Details below.
>>>>
>>>> Craig
>>>>
>>>> 1. Yahoo specifics
>>>>
>>>> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
>>>>
>>>> doHod() contains yahoo specific stuff. I'm not sure Hod has
>>>> stabalised sufficiently for this to be changed
>>>> fixUpDomain() assumes unqualified hostnames are part of the
>>>> .inktomisearch.com DNS domain
>>>>
>>>> src/org/apache/pig/impl/PigContext.java:125
>>>> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
>>>> Note - these is no Pig command to allow imports of package
>>>> namespaces into the packageImportList ArrayList
>>>>
>>>> scripts/pig.pl
>>>> kryptontite mentions, specifics: 69, 114
>>>>
>>>> 2. Extensions to Pig syntax
>>>> (a) "set" command sets all system properties
>>>> (b) "include" includes and parses another pig script
>>>> (c) "import" adds a package namespace to the search path
>>>>
>>>> 3. Change so that ~/.pigrc into a pig script that is parsed on
>>>> startup of Grunt/PigServer?
>>>>
>>>
>>
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Benjamin Francisoud <be...@joost.com>.
Alan Gates a écrit :
> It is already possible to do the following:
>
> PigContext context = new PigContext(execType); // execType is
> PigServer.ExecType
> PigServer pig = new PigServer(context);
> pig.registerQuery(...);
> ...
>
> PigContext contains, among other things, a Properties object that keep
> the properties. What's missing is a way to set values in that
> properties object. So if we added a setProperty() method to
> PigContext I think we'd have what you suggest below.
Yes I think that would provide more flexibility (at least for me :-) )
btw I have a question about using Properties class. May be I missed the
reason but java.util.Map is more generic and would allow to change
internal implementation easily doing someting like:
Map props = new Properties();
public Map getProperties(){}
public void setProperties(Map props){}
my 2 cents
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Alan Gates <ga...@yahoo-inc.com>.
It is already possible to do the following:
PigContext context = new PigContext(execType); // execType is
PigServer.ExecType
PigServer pig = new PigServer(context);
pig.registerQuery(...);
...
PigContext contains, among other things, a Properties object that keep
the properties. What's missing is a way to set values in that
properties object. So if we added a setProperty() method to PigContext
I think we'd have what you suggest below.
Alan.
Benjamin Francisoud wrote:
> I'm not a project lead but as a user of pig I'd like a possibility to
> configure pig programmatically.
>
> Doing something like:
>
> PigConfiguration conf = new PigConfiguration();
> conf.set("foo", "bar");
> Pig pig = new Pig();
> pig.setConf(conf);
> pig.run();
>
> This is just a pseudo-code example.
> Replace Pig and PigConfiguration with whatever name you like ;)
>
> --
> Benjamin Francisoud
>
> Craig Macdonald a écrit :
>> Hi pig-devs,
>>
>> Just a quiet ping for comments from project leads on this. It seems I
>> have raised several issues recently that require configuration of
>> pig. Questions:
>> * Are System properties the best place for these?
>> * Should .pigrc evolve into a place for Pig aliases and properties,
>> and even scripts? (similar to .bashrc etc)
>> * Should new commands be added: import, include, sharedFS etc?
>> * Please direct me as to how JIRAs should be created.
>> I may be able to provide patches to some JIRAs I have created if we
>> have a policy for configuration-type stuff.
>>
>> C
>>
>> Craig Macdonald wrote:
>>> Good morning Pig-devs,
>>>
>>> This email notes some of the yahoo specifics remaining in Pig that
>>> may be needed to checked before a Pig release (see 1. below). I
>>> would hope that Pig syntax can be evolved to allow these to be
>>> removed (see (2) below), and instead placed in users .pigrc.
>>>
>>> From PigContext, I note that the .pigrc is in fact a place for
>>> properties. An alternative would be for the Grunt set command to set
>>> System properties, and then make .pigrc into a pig script, allowing
>>> users to define aliases, register common jar files, import common
>>> namespaces, include other pig script files.
>>>
>>> Please direct how JIRAs should be created to track these issues -
>>> one issue for all, with subtasks; separate tasks for the three
>>> issues below?
>>>
>>> Details below.
>>>
>>> Craig
>>>
>>> 1. Yahoo specifics
>>>
>>> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
>>> doHod() contains yahoo specific stuff. I'm not sure Hod has
>>> stabalised sufficiently for this to be changed
>>> fixUpDomain() assumes unqualified hostnames are part of the
>>> .inktomisearch.com DNS domain
>>>
>>> src/org/apache/pig/impl/PigContext.java:125
>>> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
>>> Note - these is no Pig command to allow imports of package
>>> namespaces into the packageImportList ArrayList
>>>
>>> scripts/pig.pl
>>> kryptontite mentions, specifics: 69, 114
>>>
>>> 2. Extensions to Pig syntax
>>> (a) "set" command sets all system properties
>>> (b) "include" includes and parses another pig script
>>> (c) "import" adds a package namespace to the search path
>>>
>>> 3. Change so that ~/.pigrc into a pig script that is parsed on
>>> startup of Grunt/PigServer?
>>>
>>
>
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Benjamin Francisoud <be...@joost.com>.
I'm not a project lead but as a user of pig I'd like a possibility to
configure pig programmatically.
Doing something like:
PigConfiguration conf = new PigConfiguration();
conf.set("foo", "bar");
Pig pig = new Pig();
pig.setConf(conf);
pig.run();
This is just a pseudo-code example.
Replace Pig and PigConfiguration with whatever name you like ;)
--
Benjamin Francisoud
Craig Macdonald a écrit :
> Hi pig-devs,
>
> Just a quiet ping for comments from project leads on this. It seems I
> have raised several issues recently that require configuration of pig.
> Questions:
> * Are System properties the best place for these?
> * Should .pigrc evolve into a place for Pig aliases and properties,
> and even scripts? (similar to .bashrc etc)
> * Should new commands be added: import, include, sharedFS etc?
> * Please direct me as to how JIRAs should be created.
> I may be able to provide patches to some JIRAs I have created if we
> have a policy for configuration-type stuff.
>
> C
>
> Craig Macdonald wrote:
>> Good morning Pig-devs,
>>
>> This email notes some of the yahoo specifics remaining in Pig that
>> may be needed to checked before a Pig release (see 1. below). I would
>> hope that Pig syntax can be evolved to allow these to be removed (see
>> (2) below), and instead placed in users .pigrc.
>>
>> From PigContext, I note that the .pigrc is in fact a place for
>> properties. An alternative would be for the Grunt set command to set
>> System properties, and then make .pigrc into a pig script, allowing
>> users to define aliases, register common jar files, import common
>> namespaces, include other pig script files.
>>
>> Please direct how JIRAs should be created to track these issues - one
>> issue for all, with subtasks; separate tasks for the three issues below?
>>
>> Details below.
>>
>> Craig
>>
>> 1. Yahoo specifics
>>
>> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
>> doHod() contains yahoo specific stuff. I'm not sure Hod has
>> stabalised sufficiently for this to be changed
>> fixUpDomain() assumes unqualified hostnames are part of the
>> .inktomisearch.com DNS domain
>>
>> src/org/apache/pig/impl/PigContext.java:125
>> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
>> Note - these is no Pig command to allow imports of package
>> namespaces into the packageImportList ArrayList
>>
>> scripts/pig.pl
>> kryptontite mentions, specifics: 69, 114
>>
>> 2. Extensions to Pig syntax
>> (a) "set" command sets all system properties
>> (b) "include" includes and parses another pig script
>> (c) "import" adds a package namespace to the search path
>>
>> 3. Change so that ~/.pigrc into a pig script that is parsed on
>> startup of Grunt/PigServer?
>>
>
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Benjamin Francisoud <be...@joost.com>.
Craig Macdonald a écrit :
>
> (1) Why should a user have to supply the fully qualified name to his
> user defined function, if all the functions he ever uses are in that
> package? Obviously, he shouldnt have to, which is why PigContext
> includes this line:
> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
>
> I'm asking to add a command that allows me to do something like:
> import uk.ac.gla.terrier.pig
> and have that package searched for any functions. Yahoo users have
> this ability (com.yahoo.pig.yst.sds.ULT. is searched by default), why
> not everyone else ;-)
> [Somewhat similar to the define keyword.]
>
Reminds me a lot of XQuery "import module namespace" thing
http://www.w3.org/TR/xquery/#dt-module-import
some examples:
import module namespace math = "http://example.org/math-functions";
declare namespace foo = "http://example.org";
May be not stick to the same exact syntax but to get some inspiration...
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Craig Macdonald <cr...@dcs.gla.ac.uk>.
Hi Alan et al,
comments through.
> Here's my thinking on this, though I don't speak for all of the
> committers. Pig should have 3 ways to pick up configuration:
>
> 1) from .pigrc, as it does now
agreed
> 2) when embedded in another java program, the caller should be able to
> set values in PigContext, as I referred to in my response to
> Benjamin's email.
agreed
> 3) From the pig script, we should be able to something like: set
> conf.x = y (I'm not necessarily suggesting syntax here).
Ok, start of a patch for this attached.
I wasn't sure if the "conf." prefix is required, so patch has this as a
comment.
>
>> * Should .pigrc evolve into a place for Pig aliases and properties,
>> and even scripts? (similar to .bashrc etc)
> Right now you can store pig properties here. It's not clear it needs
> to grow beyond that. What use case do you see for storing aliases or
> scripts here?
(motivations below)
>> * Should new commands be added: import, include, sharedFS etc?
> I'm guessing this is the same things as I'm saying in 3 above. If
> not, please elaborate on what these new commands would do.
I'm trying to envisage how to allow reusability in the Pig.
These are mostly from my original email.
>>>
>>> 2. Extensions to Pig syntax
>>> (a) "set" command sets all system properties
>>> (b) "include" includes and parses another pig script
>>> (c) "import" adds a package namespace to the search path
>>>
>>> 3. Change so that ~/.pigrc into a pig script that is parsed on
>>> startup of Grunt/PigServer?
(1) Why should a user have to supply the fully qualified name to his
user defined function, if all the functions he ever uses are in that
package? Obviously, he shouldnt have to, which is why PigContext
includes this line:
packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
I'm asking to add a command that allows me to do something like:
import uk.ac.gla.terrier.pig
and have that package searched for any functions. Yahoo users have this
ability (com.yahoo.pig.yst.sds.ULT. is searched by default), why not
everyone else ;-)
[Somewhat similar to the define keyword.]
These could be properties instead.
(2) Include other pig files. Just to allow commonly created imports,
configuration, defines, etc to be easily loaded. How often do you
register the same jar files time-in time-out for every pig script that
your write.
(3) sharedFS - see PIG-102 - equally could be a property too.
(4) pigrc as a script - similar to (2).
This is like your Unix shell rc, eg .bashrc
Mine is full of single character aliases for commands I use all the
time, etc.
C
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Alan Gates <ga...@yahoo-inc.com>.
Craig,
Here's my thinking on this, though I don't speak for all of the
committers. Pig should have 3 ways to pick up configuration:
1) from .pigrc, as it does now
2) when embedded in another java program, the caller should be able to
set values in PigContext, as I referred to in my response to Benjamin's
email.
3) From the pig script, we should be able to something like: set conf.x
= y (I'm not necessarily suggesting syntax here).
With those in mind, to answer you're specific questions:
Craig Macdonald wrote:
> Hi pig-devs,
>
> Just a quiet ping for comments from project leads on this. It seems I
> have raised several issues recently that require configuration of pig.
> Questions:
> * Are System properties the best place for these?
We seem to be putting them in PigContext.conf for now, which is probably
a fine place for them.
> * Should .pigrc evolve into a place for Pig aliases and properties,
> and even scripts? (similar to .bashrc etc)
Right now you can store pig properties here. It's not clear it needs to
grow beyond that. What use case do you see for storing aliases or
scripts here?
> * Should new commands be added: import, include, sharedFS etc?
I'm guessing this is the same things as I'm saying in 3 above. If not,
please elaborate on what these new commands would do.
> * Please direct me as to how JIRAs should be created.
> I may be able to provide patches to some JIRAs I have created if we
> have a policy for configuration-type stuff.
>
> C
>
> Craig Macdonald wrote:
>> Good morning Pig-devs,
>>
>> This email notes some of the yahoo specifics remaining in Pig that
>> may be needed to checked before a Pig release (see 1. below). I would
>> hope that Pig syntax can be evolved to allow these to be removed (see
>> (2) below), and instead placed in users .pigrc.
>>
>> From PigContext, I note that the .pigrc is in fact a place for
>> properties. An alternative would be for the Grunt set command to set
>> System properties, and then make .pigrc into a pig script, allowing
>> users to define aliases, register common jar files, import common
>> namespaces, include other pig script files.
>>
>> Please direct how JIRAs should be created to track these issues - one
>> issue for all, with subtasks; separate tasks for the three issues below?
>>
>> Details below.
>>
>> Craig
>>
>> 1. Yahoo specifics
>>
>> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
>> doHod() contains yahoo specific stuff. I'm not sure Hod has
>> stabalised sufficiently for this to be changed
>> fixUpDomain() assumes unqualified hostnames are part of the
>> .inktomisearch.com DNS domain
>>
>> src/org/apache/pig/impl/PigContext.java:125
>> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
>> Note - these is no Pig command to allow imports of package
>> namespaces into the packageImportList ArrayList
>>
>> scripts/pig.pl
>> kryptontite mentions, specifics: 69, 114
>>
>> 2. Extensions to Pig syntax
>> (a) "set" command sets all system properties
>> (b) "include" includes and parses another pig script
>> (c) "import" adds a package namespace to the search path
>>
>> 3. Change so that ~/.pigrc into a pig script that is parsed on
>> startup of Grunt/PigServer?
>>
>
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Craig Macdonald <cr...@dcs.gla.ac.uk>.
Hi pig-devs,
Just a quiet ping for comments from project leads on this. It seems I
have raised several issues recently that require configuration of pig.
Questions:
* Are System properties the best place for these?
* Should .pigrc evolve into a place for Pig aliases and properties, and
even scripts? (similar to .bashrc etc)
* Should new commands be added: import, include, sharedFS etc?
* Please direct me as to how JIRAs should be created.
I may be able to provide patches to some JIRAs I have created if we have
a policy for configuration-type stuff.
C
Craig Macdonald wrote:
> Good morning Pig-devs,
>
> This email notes some of the yahoo specifics remaining in Pig that may
> be needed to checked before a Pig release (see 1. below). I would hope
> that Pig syntax can be evolved to allow these to be removed (see (2)
> below), and instead placed in users .pigrc.
>
> From PigContext, I note that the .pigrc is in fact a place for
> properties. An alternative would be for the Grunt set command to set
> System properties, and then make .pigrc into a pig script, allowing
> users to define aliases, register common jar files, import common
> namespaces, include other pig script files.
>
> Please direct how JIRAs should be created to track these issues - one
> issue for all, with subtasks; separate tasks for the three issues below?
>
> Details below.
>
> Craig
>
> 1. Yahoo specifics
>
> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
> doHod() contains yahoo specific stuff. I'm not sure Hod has
> stabalised sufficiently for this to be changed
> fixUpDomain() assumes unqualified hostnames are part of the
> .inktomisearch.com DNS domain
>
> src/org/apache/pig/impl/PigContext.java:125
> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
> Note - these is no Pig command to allow imports of package
> namespaces into the packageImportList ArrayList
>
> scripts/pig.pl
> kryptontite mentions, specifics: 69, 114
>
> 2. Extensions to Pig syntax
> (a) "set" command sets all system properties
> (b) "include" includes and parses another pig script
> (c) "import" adds a package namespace to the search path
>
> 3. Change so that ~/.pigrc into a pig script that is parsed on startup
> of Grunt/PigServer?
>
Re: yahoo-specific pig - improvements to syntax and .pigrc
Posted by Benjamin Francisoud <be...@joost.com>.
There is also SSHSocketImplFactory line:449
connectTest("www.yahoo.com");
There is also www.news.com and www.flickr.com in the same classe.
I don't know what this SSHSocketImplFactory #main() method is use for...
Craig Macdonald a écrit :
> Good morning Pig-devs,
>
> This email notes some of the yahoo specifics remaining in Pig that may
> be needed to checked before a Pig release (see 1. below). I would hope
> that Pig syntax can be evolved to allow these to be removed (see (2)
> below), and instead placed in users .pigrc.
>
> From PigContext, I note that the .pigrc is in fact a place for
> properties. An alternative would be for the Grunt set command to set
> System properties, and then make .pigrc into a pig script, allowing
> users to define aliases, register common jar files, import common
> namespaces, include other pig script files.
>
> Please direct how JIRAs should be created to track these issues - one
> issue for all, with subtasks; separate tasks for the three issues below?
>
> Details below.
>
> Craig
>
> 1. Yahoo specifics
>
> src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
> doHod() contains yahoo specific stuff. I'm not sure Hod has
> stabalised sufficiently for this to be changed
> fixUpDomain() assumes unqualified hostnames are part of the
> .inktomisearch.com DNS domain
>
> src/org/apache/pig/impl/PigContext.java:125
> packageImportList.add("com.yahoo.pig.yst.sds.ULT.");
> Note - these is no Pig command to allow imports of package
> namespaces into the packageImportList ArrayList
>
> scripts/pig.pl
> kryptontite mentions, specifics: 69, 114
>
> 2. Extensions to Pig syntax
> (a) "set" command sets all system properties
> (b) "include" includes and parses another pig script
> (c) "import" adds a package namespace to the search path
>
> 3. Change so that ~/.pigrc into a pig script that is parsed on startup
> of Grunt/PigServer?
>