You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pig.apache.org by Apache Wiki <wi...@apache.org> on 2009/04/15 18:27:49 UTC
[Pig Wiki] Update of "ProposedProjects" by AlanGates
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by AlanGates:
http://wiki.apache.org/pig/ProposedProjects
New page:
= Proposed Pig Projects =
This page describes projects what we (the committers) would like to see added
to Pig. The scale of these projects vary, but they are larger projects,
usually on the weeks or months scale. We have not yet filed
[https://issues.apache.org/jira/browse/PIG JIRAs] for some of these
because they are still in the vague idea stage. As they become more concrete,
[https://issues.apache.org/jira/browse/PIG JIRAs] will be filed for them.
We welcome contributers to take on one of these projects. If you would like
to do so, please file a JIRA (if one does not already exist for the project)
with a proposed solution. Pig's committers will work with you from there to
help refine your solution. Once a solution is agreed upon, you can begin
implementation.
If you see a project here that you would like to see Pig implement but you are
not in a position to implement the solution right now, feel free to vote for
the project. Add your name to the list of supporters. This will help
contributers looking for a project to select one that will benefit many users.
If you would like to propose a project for Pig, feel free to add to this list.
If it is a smaller project, or something you plan to begin work on
immediately, filing a [https://issues.apache.org/jira/browse/PIG JIRA] is a better route.
|| Catagory || Project || JIRA || Proposed By || Votes For ||
|| Execution || Pig currently executes scripts by building a pipeline of pre-built operators and running data through those operators in map reduce jobs. We need to investigate instead have Pig generate java code specific to a job, and then compiling that code and using it to run the map reduce jobs. || || Many conference attendees || gates ||
|| Language || Currently only DISTINCT, ORDER BY, and FILTER are allowed inside FOREACH. All operators should be allowed in FOREACH. (Limit is being worked on [https://issues.apache.org/jira/browse/PIG-741 741] || || gates || ||
|| Optimization || Speed up comparison of tuples during shuffle for ORDER BY || [https://issues.apache.org/jira/browse/PIG-659 659] || olgan || ||
|| Optimization || Order by should be changed to not use POPackage to put all of the tuples in a bag on the reduce side, as the bag is just immediately flattened. It can instead work like join does for the last input in the join. || || gates || ||
|| Optimization || Often in a Pig script that produces a chain of MR jobs, the map phases of 2nd and subsequent jobs very little. What little they do should be pushed into the proceeding reduce and the map replaced by the identity mapper. Initial tests showed that the identity mapper was 50% faster than using a Pig mapper (because Pig uses the loader to parse out tuples even if the map itself is empty). || [https://issues.apache.org/jira/browse/PIG-480 480] || olgan || gates ||
|| Optimization || Use hand crafted calls to do string to integer or float conversions. Initial tests showed these could be done about 8x faster than String.toIntger() and String.toFloat(). || [https://issues.apache.org/jira/browse/PIG-482 482] || olgan || gates ||
|| Optimization || Currently Pig always samples for and ORDER BY to determine how to partition, and then runs another job to do the sort. For small enough inputs, it should just sort with a single reducer. || [https://issues.apache.org/jira/browse/PIG-483 483] || olgan || ||
|| Optimization || In many cases data to be joined is already sorted and partitioned on the same key. Pig needs to be able to take advantage of this and do these joins in the map. The join could be done by sampling one input to determine the value of the join key at the beginning of every HDFS block. This would form an index. Then in a second MR job can be run with the other input. Based on the key seen in the second input, the appropriate blocks of the first input can also be loaded into the map and the join done. || || gates || ||
|| Optimization || The combiner is not currently used if FILTER is in the FOREACH. In some cases it could still be used. || [https://issues.apache.org/jira/browse/PIG-479 479] || olgan || ||
|| Optimization || Currently when types of data are declared Pig inserts a FOREACH immediately after the LOAD that does the conversions. These conversions should be delayed until the field is actually used. || [https://issues.apache.org/jira/browse/PIG-410 410] || olgan || gates ||
|| Optimization || When an order by is not the only operation in a pig script, it is done in two additional MR jobs. The first job samples using a sampling loader, the second does the sort. The sample is used to construct a partitioner that equally balances the data in the sort. The sampler needs to be changed to be a !EvalFunc instead of a loader. This way a split can be but in the proceeding MR job, with the main data being written out and the other part flowing to the sampler func, which can then write out the sample. The final MR job can then be the sort. || || gates || ||
|| Optimization || When an order by is the only operation in a pig script it is currently done in 3 MR jobs. The first converts it to BinStorage format (because the sample loader reads that format), the second samples, and the third sorts. Once the changes mentioned above to make the sampler an !EvalFunc are done it should be changed to be done in 2 MR jobs instead of 3. || [https://issues.apache.org/jira/browse/PIG-460 460] || gates || ||
|| Optimization || The Pig optimizer should be used to determine when fields in a record are no longer needed and put in FOREACH statements to project out the unecessary data as early as possible. || [https://issues.apache.org/jira/browse/PIG-466 466] || olgan || ||
|| Optimization || The Pig optimizers needs to call fieldsToRead so that Load functions that can do column skipping do it. || || gates || ||
|| Scalability || Pig's default join (symmetric hash) currently depends on being able to fit all of the values for a given join key for one of the inputs into memory. (It does try to spill to disk in the case where it cannot fit them all into memory. In practice this often fails as it is not good at understanding when memory is low enough that it should spill. Even in the case where it does not fail, spilling to disk and rereading from disk is very slow.) If instances of keys with a large number of values were broken up so that the row set could fit in memory and then shipped to multiple reducers. A sampling pass would need to be done first to determine which keys to break up. || || chris olston || gates ||
Re: [Pig Wiki] Update of "ProposedProjects" by AlanGates
Posted by Thejas Nair <te...@yahoo-inc.com>.
This paper seems very relevant to the proposal -
"Compiled Query Execution Engine using JVM"
http://www2.computer.org/portal/web/csdl/doi/10.1109/ICDE.2006.40
>From the abstract -
"Our experimental results on the TPC-H data set show that, despite both
engines benefiting from JIT, the compiled engine runs on average about twice
as fast as the interpreted one, and significantly faster than an in-memory"
(I don't have access to the full paper though).
-Thejas
On 4/16/09 9:26 AM, "Alan Gates" <ga...@yahoo-inc.com> wrote:
> Your understanding of the proposal is correct. The goal would be to
> produce Java code rather than a pipeline configuration. But the
> reasoning is not so that users can then take that and modify
> themselves. There's nothing preventing them from doing it, but it has
> a couple of major drawbacks.
>
> 1) Code generators generally generate horrific looking code, because
> they are going for speed and compactness not human maintainability.
> Trying to work in that code would be very difficult.
>
> 2) If you start adding code to generated code, you can no longer use
> the original Pig Latin. You are from that point forward stuck in
> Java, since you can't backport your Java into the Pig Latin.
>
> The proposal is designed to test the performance of Pig based on
> generated Java (or for that matter any other language, it need not be
> Java). For the idea you suggest, the NATIVE keyword (proposed here
> https://issues.apache.org/jira/browse/PIG-506)
> is a better solution.
>
> Alan.
>
> On Apr 16, 2009, at 12:54 AM, nitesh bhatia wrote:
>
>> Hi
>> Can you briefly explain what is required in the first project? After
>> reading
>> the description my impression is, currently when we are executing
>> commands
>> on Pig Shell, Pig is first converting to map-reduce jobs and then
>> feeding it
>> to hadoop. In this project are we proposing that, the execution plan
>> made by
>> Pig will be first converted to a java file for map-reduce procedure
>> and then
>> feed onto hadoop network ?
>>
>> If this is the case then I am sure it will be great help to users as
>> this
>> functionality can be used to write complicated map-reduce jobs very
>> easily.
>> Initially user can write the Pig scripts / commands required for his
>> job and
>> get the map-reduce java files. Then he can edit map-reduce files to
>> extend
>> the functionality and add extra procedures that are not provided by
>> Pig but
>> can be executed over hadoop.
>>
>> --nitesh
>>
>> On Wed, Apr 15, 2009 at 9:57 PM, Apache Wiki <wi...@apache.org>
>> wrote:
>>
>>> Dear Wiki user,
>>>
>>> You have subscribed to a wiki page or wiki category on "Pig Wiki" for
>>> change notification.
>>>
>>> The following page has been changed by AlanGates:
>>> http://wiki.apache.org/pig/ProposedProjects
>>>
>>> New page:
>>> = Proposed Pig Projects =
>>> This page describes projects what we (the committers) would like to
>>> see
>>> added
>>> to Pig. The scale of these projects vary, but they are larger
>>> projects,
>>> usually on the weeks or months scale. We have not yet filed
>>> [https://issues.apache.org/jira/browse/PIG JIRAs] for some of these
>>> because they are still in the vague idea stage. As they become more
>>> concrete,
>>> [https://issues.apache.org/jira/browse/PIG JIRAs] will be filed for
>>> them.
>>>
>>> We welcome contributers to take on one of these projects. If you
>>> would
>>> like
>>> to do so, please file a JIRA (if one does not already exist for the
>>> project)
>>> with a proposed solution. Pig's committers will work with you from
>>> there
>>> to
>>> help refine your solution. Once a solution is agreed upon, you can
>>> begin
>>> implementation.
>>>
>>> If you see a project here that you would like to see Pig implement
>>> but you
>>> are
>>> not in a position to implement the solution right now, feel free to
>>> vote
>>> for
>>> the project. Add your name to the list of supporters. This will
>>> help
>>> contributers looking for a project to select one that will benefit
>>> many
>>> users.
>>>
>>> If you would like to propose a project for Pig, feel free to add to
>>> this
>>> list.
>>> If it is a smaller project, or something you plan to begin work on
>>> immediately, filing a [https://issues.apache.org/jira/browse/PIG
>>> JIRA] is
>>> a better route.
>>>
>>> || Catagory || Project || JIRA || Proposed By || Votes For ||
>>> || Execution || Pig currently executes scripts by building a
>>> pipeline of
>>> pre-built operators and running data through those operators in map
>>> reduce
>>> jobs. We need to investigate instead have Pig generate java code
>>> specific
>>> to a job, and then compiling that code and using it to run the map
>>> reduce
>>> jobs. || || Many conference attendees || gates ||
>>> || Language || Currently only DISTINCT, ORDER BY, and FILTER are
>>> allowed
>>> inside FOREACH. All operators should be allowed in FOREACH. (Limit
>>> is being
>>> worked on [https://issues.apache.org/jira/browse/PIG-741 741] || ||
>>> gates
>>> || ||
>>> || Optimization || Speed up comparison of tuples during shuffle for
>>> ORDER
>>> BY || [https://issues.apache.org/jira/browse/PIG-659 659] || olgan
>>> || ||
>>> || Optimization || Order by should be changed to not use POPackage
>>> to put
>>> all of the tuples in a bag on the reduce side, as the bag is just
>>> immediately flattened. It can instead work like join does for the
>>> last
>>> input in the join. || || gates || ||
>>> || Optimization || Often in a Pig script that produces a chain of
>>> MR jobs,
>>> the map phases of 2nd and subsequent jobs very little. What little
>>> they do
>>> should be pushed into the proceeding reduce and the map replaced by
>>> the
>>> identity mapper. Initial tests showed that the identity mapper was
>>> 50%
>>> faster than using a Pig mapper (because Pig uses the loader to
>>> parse out
>>> tuples even if the map itself is empty). || [
>>> https://issues.apache.org/jira/browse/PIG-480 480] || olgan ||
>>> gates ||
>>> || Optimization || Use hand crafted calls to do string to integer
>>> or float
>>> conversions. Initial tests showed these could be done about 8x
>>> faster than
>>> String.toIntger() and String.toFloat(). || [
>>> https://issues.apache.org/jira/browse/PIG-482 482] || olgan ||
>>> gates ||
>>> || Optimization || Currently Pig always samples for and ORDER BY to
>>> determine how to partition, and then runs another job to do the
>>> sort. For
>>> small enough inputs, it should just sort with a single reducer. || [
>>> https://issues.apache.org/jira/browse/PIG-483 483] || olgan || ||
>>> || Optimization || In many cases data to be joined is already
>>> sorted and
>>> partitioned on the same key. Pig needs to be able to take
>>> advantage of this
>>> and do these joins in the map. The join could be done by sampling
>>> one input
>>> to determine the value of the join key at the beginning of every
>>> HDFS block.
>>> This would form an index. Then in a second MR job can be run with
>>> the
>>> other input. Based on the key seen in the second input, the
>>> appropriate
>>> blocks of the first input can also be loaded into the map and the
>>> join done.
>>> || || gates || ||
>>> || Optimization || The combiner is not currently used if FILTER is
>>> in the
>>> FOREACH. In some cases it could still be used. || [
>>> https://issues.apache.org/jira/browse/PIG-479 479] || olgan || ||
>>> || Optimization || Currently when types of data are declared Pig
>>> inserts a
>>> FOREACH immediately after the LOAD that does the conversions. These
>>> conversions should be delayed until the field is actually used. || [
>>> https://issues.apache.org/jira/browse/PIG-410 410] || olgan ||
>>> gates ||
>>> || Optimization || When an order by is not the only operation in a
>>> pig
>>> script, it is done in two additional MR jobs. The first job
>>> samples using a
>>> sampling loader, the second does the sort. The sample is used to
>>> construct
>>> a partitioner that equally balances the data in the sort. The
>>> sampler needs
>>> to be changed to be a !EvalFunc instead of a loader. This way a
>>> split can
>>> be but in the proceeding MR job, with the main data being written
>>> out and
>>> the other part flowing to the sampler func, which can then write
>>> out the
>>> sample. The final MR job can then be the sort. || || gates || ||
>>> || Optimization || When an order by is the only operation in a pig
>>> script
>>> it is currently done in 3 MR jobs. The first converts it to
>>> BinStorage
>>> format (because the sample loader reads that format), the second
>>> samples,
>>> and the third sorts. Once the changes mentioned above to make the
>>> sampler
>>> an !EvalFunc are done it should be changed to be done in 2 MR jobs
>>> instead
>>> of 3. || [https://issues.apache.org/jira/browse/PIG-460 460] ||
>>> gates ||
>>> ||
>>> || Optimization || The Pig optimizer should be used to determine when
>>> fields in a record are no longer needed and put in FOREACH
>>> statements to
>>> project out the unecessary data as early as possible. || [
>>> https://issues.apache.org/jira/browse/PIG-466 466] || olgan || ||
>>> || Optimization || The Pig optimizers needs to call fieldsToRead so
>>> that
>>> Load functions that can do column skipping do it. || || gates || ||
>>> || Scalability || Pig's default join (symmetric hash) currently
>>> depends on
>>> being able to fit all of the values for a given join key for one of
>>> the
>>> inputs into memory. (It does try to spill to disk in the case
>>> where it
>>> cannot fit them all into memory. In practice this often fails as
>>> it is not
>>> good at understanding when memory is low enough that it should
>>> spill. Even
>>> in the case where it does not fail, spilling to disk and rereading
>>> from disk
>>> is very slow.) If instances of keys with a large number of values
>>> were
>>> broken up so that the row set could fit in memory and then shipped to
>>> multiple reducers. A sampling pass would need to be done first to
>>> determine
>>> which keys to break up. || || chris olston || gates ||
>>>
>>
>>
>>
>> --
>> Nitesh Bhatia
>> Dhirubhai Ambani Institute of Information & Communication Technology
>> Gandhinagar
>> Gujarat
>>
>> "Life is never perfect. It just depends where you draw the line."
>>
>> visit:
>> http://www.awaaaz.com - connecting through music
>> http://www.volstreet.com - lets volunteer for better tomorrow
>> http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
>
Re: [Pig Wiki] Update of "ProposedProjects" by AlanGates
Posted by Alan Gates <ga...@yahoo-inc.com>.
Your understanding of the proposal is correct. The goal would be to
produce Java code rather than a pipeline configuration. But the
reasoning is not so that users can then take that and modify
themselves. There's nothing preventing them from doing it, but it has
a couple of major drawbacks.
1) Code generators generally generate horrific looking code, because
they are going for speed and compactness not human maintainability.
Trying to work in that code would be very difficult.
2) If you start adding code to generated code, you can no longer use
the original Pig Latin. You are from that point forward stuck in
Java, since you can't backport your Java into the Pig Latin.
The proposal is designed to test the performance of Pig based on
generated Java (or for that matter any other language, it need not be
Java). For the idea you suggest, the NATIVE keyword (proposed here https://issues.apache.org/jira/browse/PIG-506)
is a better solution.
Alan.
On Apr 16, 2009, at 12:54 AM, nitesh bhatia wrote:
> Hi
> Can you briefly explain what is required in the first project? After
> reading
> the description my impression is, currently when we are executing
> commands
> on Pig Shell, Pig is first converting to map-reduce jobs and then
> feeding it
> to hadoop. In this project are we proposing that, the execution plan
> made by
> Pig will be first converted to a java file for map-reduce procedure
> and then
> feed onto hadoop network ?
>
> If this is the case then I am sure it will be great help to users as
> this
> functionality can be used to write complicated map-reduce jobs very
> easily.
> Initially user can write the Pig scripts / commands required for his
> job and
> get the map-reduce java files. Then he can edit map-reduce files to
> extend
> the functionality and add extra procedures that are not provided by
> Pig but
> can be executed over hadoop.
>
> --nitesh
>
> On Wed, Apr 15, 2009 at 9:57 PM, Apache Wiki <wi...@apache.org>
> wrote:
>
>> Dear Wiki user,
>>
>> You have subscribed to a wiki page or wiki category on "Pig Wiki" for
>> change notification.
>>
>> The following page has been changed by AlanGates:
>> http://wiki.apache.org/pig/ProposedProjects
>>
>> New page:
>> = Proposed Pig Projects =
>> This page describes projects what we (the committers) would like to
>> see
>> added
>> to Pig. The scale of these projects vary, but they are larger
>> projects,
>> usually on the weeks or months scale. We have not yet filed
>> [https://issues.apache.org/jira/browse/PIG JIRAs] for some of these
>> because they are still in the vague idea stage. As they become more
>> concrete,
>> [https://issues.apache.org/jira/browse/PIG JIRAs] will be filed for
>> them.
>>
>> We welcome contributers to take on one of these projects. If you
>> would
>> like
>> to do so, please file a JIRA (if one does not already exist for the
>> project)
>> with a proposed solution. Pig's committers will work with you from
>> there
>> to
>> help refine your solution. Once a solution is agreed upon, you can
>> begin
>> implementation.
>>
>> If you see a project here that you would like to see Pig implement
>> but you
>> are
>> not in a position to implement the solution right now, feel free to
>> vote
>> for
>> the project. Add your name to the list of supporters. This will
>> help
>> contributers looking for a project to select one that will benefit
>> many
>> users.
>>
>> If you would like to propose a project for Pig, feel free to add to
>> this
>> list.
>> If it is a smaller project, or something you plan to begin work on
>> immediately, filing a [https://issues.apache.org/jira/browse/PIG
>> JIRA] is
>> a better route.
>>
>> || Catagory || Project || JIRA || Proposed By || Votes For ||
>> || Execution || Pig currently executes scripts by building a
>> pipeline of
>> pre-built operators and running data through those operators in map
>> reduce
>> jobs. We need to investigate instead have Pig generate java code
>> specific
>> to a job, and then compiling that code and using it to run the map
>> reduce
>> jobs. || || Many conference attendees || gates ||
>> || Language || Currently only DISTINCT, ORDER BY, and FILTER are
>> allowed
>> inside FOREACH. All operators should be allowed in FOREACH. (Limit
>> is being
>> worked on [https://issues.apache.org/jira/browse/PIG-741 741] || ||
>> gates
>> || ||
>> || Optimization || Speed up comparison of tuples during shuffle for
>> ORDER
>> BY || [https://issues.apache.org/jira/browse/PIG-659 659] || olgan
>> || ||
>> || Optimization || Order by should be changed to not use POPackage
>> to put
>> all of the tuples in a bag on the reduce side, as the bag is just
>> immediately flattened. It can instead work like join does for the
>> last
>> input in the join. || || gates || ||
>> || Optimization || Often in a Pig script that produces a chain of
>> MR jobs,
>> the map phases of 2nd and subsequent jobs very little. What little
>> they do
>> should be pushed into the proceeding reduce and the map replaced by
>> the
>> identity mapper. Initial tests showed that the identity mapper was
>> 50%
>> faster than using a Pig mapper (because Pig uses the loader to
>> parse out
>> tuples even if the map itself is empty). || [
>> https://issues.apache.org/jira/browse/PIG-480 480] || olgan ||
>> gates ||
>> || Optimization || Use hand crafted calls to do string to integer
>> or float
>> conversions. Initial tests showed these could be done about 8x
>> faster than
>> String.toIntger() and String.toFloat(). || [
>> https://issues.apache.org/jira/browse/PIG-482 482] || olgan ||
>> gates ||
>> || Optimization || Currently Pig always samples for and ORDER BY to
>> determine how to partition, and then runs another job to do the
>> sort. For
>> small enough inputs, it should just sort with a single reducer. || [
>> https://issues.apache.org/jira/browse/PIG-483 483] || olgan || ||
>> || Optimization || In many cases data to be joined is already
>> sorted and
>> partitioned on the same key. Pig needs to be able to take
>> advantage of this
>> and do these joins in the map. The join could be done by sampling
>> one input
>> to determine the value of the join key at the beginning of every
>> HDFS block.
>> This would form an index. Then in a second MR job can be run with
>> the
>> other input. Based on the key seen in the second input, the
>> appropriate
>> blocks of the first input can also be loaded into the map and the
>> join done.
>> || || gates || ||
>> || Optimization || The combiner is not currently used if FILTER is
>> in the
>> FOREACH. In some cases it could still be used. || [
>> https://issues.apache.org/jira/browse/PIG-479 479] || olgan || ||
>> || Optimization || Currently when types of data are declared Pig
>> inserts a
>> FOREACH immediately after the LOAD that does the conversions. These
>> conversions should be delayed until the field is actually used. || [
>> https://issues.apache.org/jira/browse/PIG-410 410] || olgan ||
>> gates ||
>> || Optimization || When an order by is not the only operation in a
>> pig
>> script, it is done in two additional MR jobs. The first job
>> samples using a
>> sampling loader, the second does the sort. The sample is used to
>> construct
>> a partitioner that equally balances the data in the sort. The
>> sampler needs
>> to be changed to be a !EvalFunc instead of a loader. This way a
>> split can
>> be but in the proceeding MR job, with the main data being written
>> out and
>> the other part flowing to the sampler func, which can then write
>> out the
>> sample. The final MR job can then be the sort. || || gates || ||
>> || Optimization || When an order by is the only operation in a pig
>> script
>> it is currently done in 3 MR jobs. The first converts it to
>> BinStorage
>> format (because the sample loader reads that format), the second
>> samples,
>> and the third sorts. Once the changes mentioned above to make the
>> sampler
>> an !EvalFunc are done it should be changed to be done in 2 MR jobs
>> instead
>> of 3. || [https://issues.apache.org/jira/browse/PIG-460 460] ||
>> gates ||
>> ||
>> || Optimization || The Pig optimizer should be used to determine when
>> fields in a record are no longer needed and put in FOREACH
>> statements to
>> project out the unecessary data as early as possible. || [
>> https://issues.apache.org/jira/browse/PIG-466 466] || olgan || ||
>> || Optimization || The Pig optimizers needs to call fieldsToRead so
>> that
>> Load functions that can do column skipping do it. || || gates || ||
>> || Scalability || Pig's default join (symmetric hash) currently
>> depends on
>> being able to fit all of the values for a given join key for one of
>> the
>> inputs into memory. (It does try to spill to disk in the case
>> where it
>> cannot fit them all into memory. In practice this often fails as
>> it is not
>> good at understanding when memory is low enough that it should
>> spill. Even
>> in the case where it does not fail, spilling to disk and rereading
>> from disk
>> is very slow.) If instances of keys with a large number of values
>> were
>> broken up so that the row set could fit in memory and then shipped to
>> multiple reducers. A sampling pass would need to be done first to
>> determine
>> which keys to break up. || || chris olston || gates ||
>>
>
>
>
> --
> Nitesh Bhatia
> Dhirubhai Ambani Institute of Information & Communication Technology
> Gandhinagar
> Gujarat
>
> "Life is never perfect. It just depends where you draw the line."
>
> visit:
> http://www.awaaaz.com - connecting through music
> http://www.volstreet.com - lets volunteer for better tomorrow
> http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
Re: [Pig Wiki] Update of "ProposedProjects" by AlanGates
Posted by nitesh bhatia <ni...@gmail.com>.
Hi
Can you briefly explain what is required in the first project? After reading
the description my impression is, currently when we are executing commands
on Pig Shell, Pig is first converting to map-reduce jobs and then feeding it
to hadoop. In this project are we proposing that, the execution plan made by
Pig will be first converted to a java file for map-reduce procedure and then
feed onto hadoop network ?
If this is the case then I am sure it will be great help to users as this
functionality can be used to write complicated map-reduce jobs very easily.
Initially user can write the Pig scripts / commands required for his job and
get the map-reduce java files. Then he can edit map-reduce files to extend
the functionality and add extra procedures that are not provided by Pig but
can be executed over hadoop.
--nitesh
On Wed, Apr 15, 2009 at 9:57 PM, Apache Wiki <wi...@apache.org> wrote:
> Dear Wiki user,
>
> You have subscribed to a wiki page or wiki category on "Pig Wiki" for
> change notification.
>
> The following page has been changed by AlanGates:
> http://wiki.apache.org/pig/ProposedProjects
>
> New page:
> = Proposed Pig Projects =
> This page describes projects what we (the committers) would like to see
> added
> to Pig. The scale of these projects vary, but they are larger projects,
> usually on the weeks or months scale. We have not yet filed
> [https://issues.apache.org/jira/browse/PIG JIRAs] for some of these
> because they are still in the vague idea stage. As they become more
> concrete,
> [https://issues.apache.org/jira/browse/PIG JIRAs] will be filed for them.
>
> We welcome contributers to take on one of these projects. If you would
> like
> to do so, please file a JIRA (if one does not already exist for the
> project)
> with a proposed solution. Pig's committers will work with you from there
> to
> help refine your solution. Once a solution is agreed upon, you can begin
> implementation.
>
> If you see a project here that you would like to see Pig implement but you
> are
> not in a position to implement the solution right now, feel free to vote
> for
> the project. Add your name to the list of supporters. This will help
> contributers looking for a project to select one that will benefit many
> users.
>
> If you would like to propose a project for Pig, feel free to add to this
> list.
> If it is a smaller project, or something you plan to begin work on
> immediately, filing a [https://issues.apache.org/jira/browse/PIG JIRA] is
> a better route.
>
> || Catagory || Project || JIRA || Proposed By || Votes For ||
> || Execution || Pig currently executes scripts by building a pipeline of
> pre-built operators and running data through those operators in map reduce
> jobs. We need to investigate instead have Pig generate java code specific
> to a job, and then compiling that code and using it to run the map reduce
> jobs. || || Many conference attendees || gates ||
> || Language || Currently only DISTINCT, ORDER BY, and FILTER are allowed
> inside FOREACH. All operators should be allowed in FOREACH. (Limit is being
> worked on [https://issues.apache.org/jira/browse/PIG-741 741] || || gates
> || ||
> || Optimization || Speed up comparison of tuples during shuffle for ORDER
> BY || [https://issues.apache.org/jira/browse/PIG-659 659] || olgan || ||
> || Optimization || Order by should be changed to not use POPackage to put
> all of the tuples in a bag on the reduce side, as the bag is just
> immediately flattened. It can instead work like join does for the last
> input in the join. || || gates || ||
> || Optimization || Often in a Pig script that produces a chain of MR jobs,
> the map phases of 2nd and subsequent jobs very little. What little they do
> should be pushed into the proceeding reduce and the map replaced by the
> identity mapper. Initial tests showed that the identity mapper was 50%
> faster than using a Pig mapper (because Pig uses the loader to parse out
> tuples even if the map itself is empty). || [
> https://issues.apache.org/jira/browse/PIG-480 480] || olgan || gates ||
> || Optimization || Use hand crafted calls to do string to integer or float
> conversions. Initial tests showed these could be done about 8x faster than
> String.toIntger() and String.toFloat(). || [
> https://issues.apache.org/jira/browse/PIG-482 482] || olgan || gates ||
> || Optimization || Currently Pig always samples for and ORDER BY to
> determine how to partition, and then runs another job to do the sort. For
> small enough inputs, it should just sort with a single reducer. || [
> https://issues.apache.org/jira/browse/PIG-483 483] || olgan || ||
> || Optimization || In many cases data to be joined is already sorted and
> partitioned on the same key. Pig needs to be able to take advantage of this
> and do these joins in the map. The join could be done by sampling one input
> to determine the value of the join key at the beginning of every HDFS block.
> This would form an index. Then in a second MR job can be run with the
> other input. Based on the key seen in the second input, the appropriate
> blocks of the first input can also be loaded into the map and the join done.
> || || gates || ||
> || Optimization || The combiner is not currently used if FILTER is in the
> FOREACH. In some cases it could still be used. || [
> https://issues.apache.org/jira/browse/PIG-479 479] || olgan || ||
> || Optimization || Currently when types of data are declared Pig inserts a
> FOREACH immediately after the LOAD that does the conversions. These
> conversions should be delayed until the field is actually used. || [
> https://issues.apache.org/jira/browse/PIG-410 410] || olgan || gates ||
> || Optimization || When an order by is not the only operation in a pig
> script, it is done in two additional MR jobs. The first job samples using a
> sampling loader, the second does the sort. The sample is used to construct
> a partitioner that equally balances the data in the sort. The sampler needs
> to be changed to be a !EvalFunc instead of a loader. This way a split can
> be but in the proceeding MR job, with the main data being written out and
> the other part flowing to the sampler func, which can then write out the
> sample. The final MR job can then be the sort. || || gates || ||
> || Optimization || When an order by is the only operation in a pig script
> it is currently done in 3 MR jobs. The first converts it to BinStorage
> format (because the sample loader reads that format), the second samples,
> and the third sorts. Once the changes mentioned above to make the sampler
> an !EvalFunc are done it should be changed to be done in 2 MR jobs instead
> of 3. || [https://issues.apache.org/jira/browse/PIG-460 460] || gates ||
> ||
> || Optimization || The Pig optimizer should be used to determine when
> fields in a record are no longer needed and put in FOREACH statements to
> project out the unecessary data as early as possible. || [
> https://issues.apache.org/jira/browse/PIG-466 466] || olgan || ||
> || Optimization || The Pig optimizers needs to call fieldsToRead so that
> Load functions that can do column skipping do it. || || gates || ||
> || Scalability || Pig's default join (symmetric hash) currently depends on
> being able to fit all of the values for a given join key for one of the
> inputs into memory. (It does try to spill to disk in the case where it
> cannot fit them all into memory. In practice this often fails as it is not
> good at understanding when memory is low enough that it should spill. Even
> in the case where it does not fail, spilling to disk and rereading from disk
> is very slow.) If instances of keys with a large number of values were
> broken up so that the row set could fit in memory and then shipped to
> multiple reducers. A sampling pass would need to be done first to determine
> which keys to break up. || || chris olston || gates ||
>
--
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat
"Life is never perfect. It just depends where you draw the line."
visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun