[jira] Updated: (PIG-162) Rework mapreduce submission and monitoring

["Shravan Matthur Narayanamurthy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shravan Matthur Narayanamurthy updated PIG-162:
-----------------------------------------------

    Attachment: changes.patch

Includes some changes that have accumulated over time. Most of them are fixes to get the end to end tests working. Some others are bugs I found while running through the code and some are changes to the name() methods of the operators so that we can debug better using the PlanPrinter. I have also modified POUserFunc & some others which hold transient references to function specs more efficient by removing the if(func==null) checks. I don't know why it did not strike me then but the soln is simple. I have just overridden the readObject method of the serialization framework and after calling a default read method, I instantiate the func from the spec. So whenever other methods are called they can be sure that they will get a non null func handle. To give an overview of the changes here is a list:
1) Byte Comparison
2) Launcher separation
3) Make basic types, Cast & Sort serializable
4) Combiner plus Combiner support during JobCreation - Though I don't have test cases for this one, I am pretty confident that it gets launched. I was trying to get a hand crafted case working. But the limitations of the generate and chained projects did not enable me to do so. However, I saw through manually at least that the combiner gets called and if used appropriately, it shoud not have any problems.
5) Changes to translator that I had done to get some of the test cases working


> Rework mapreduce submission and monitoring
> ------------------------------------------
>
>                 Key: PIG-162
>                 URL: https://issues.apache.org/jira/browse/PIG-162
>             Project: Pig
>          Issue Type: Sub-task
>         Environment: This bug tracks works to rework the submission and monitoring interface to map reduce as described in  http://wiki.apache.org/pig/PigTypesFunctionalSpec
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: changes.patch, changes1.patch, incr7.patch, incr8.patch, mapreduceJumbo.patch, mapreduceJumboWithComInc.patch, modJumbo.patch, split.png, TEST-org.apache.pig.test.TestMapReduce.txt, TEST-org.apache.pig.test.TestMRCompiler.txt, TEST-org.apache.pig.test.TestMRCompiler.txt, TEST-org.apache.pig.test.TestUnion.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.