You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Olga Natkovich (JIRA)" <ji...@apache.org> on 2009/01/27 21:36:59 UTC
[jira] Created: (PIG-641) Fragment replicate join does not work in
local mode
Fragment replicate join does not work in local mode
---------------------------------------------------
Key: PIG-641
URL: https://issues.apache.org/jira/browse/PIG-641
Project: Pig
Issue Type: Bug
Reporter: Olga Natkovich
Assignee: Shubham Chopra
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-641) Fragment replicate join does not work
in local mode
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672318#action_12672318 ]
Alan Gates commented on PIG-641:
--------------------------------
Why do we need a new POFRJoinLocal operator? Can't the solution be that a logical plan of:
{code}
X Y
| |
\ /
|
FRJoin
{code}
gets translated to a physical plan of:
{code}
X Y
| |
\ /
|
Join
{code}
(that is, just a regular join) in local mode? Is there any reason to have a special operator?
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch, 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-641) Fragment replicate join does not work
in local mode
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679379#action_12679379 ]
Olga Natkovich commented on PIG-641:
------------------------------------
Shubham, could you, please, reply to Alan's question, thanks.
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch, 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-641) Fragment replicate join does not work
in local mode
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672029#action_12672029 ]
Alan Gates commented on PIG-641:
--------------------------------
The general approach here of rewriting fr join into a standard join in local mode is fine. But the parser is the wrong place to do this. We don't want to be doing backend specific operations during parsing. This should be moved to the local logical to physical translator.
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-641) Fragment replicate join does not work
in local mode
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672349#action_12672349 ]
Alan Gates commented on PIG-641:
--------------------------------
So why can't FRJoin be written to Cogroup + Foreach as well? I don't see the advantage of adding a new operator that will do exactly the same thing as two existing operators put together.
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch, 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-641) Fragment replicate join does not work in
local mode
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath resolved PIG-641.
--------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
This issue is fixed in current trunk since LocalLogToPhyTranslationVisitor always translates LOJoin into POCogroup followed by foreach flatten regardless of join type.
Here is a script I tried to validate:
[pradeepk@chargesize:~/dev/pig-apache/pig/trunk]cat a.txt
1 2 3
2 3 4
3 4 5
[pradeepk@chargesize:~/dev/pig-apache/pig/trunk]cat b.txt
3 a
1 x
4 b
[pradeepk@chargesize:~/dev/pig-apache/pig/trunk]cat c.txt
1 20 30
[pradeepk@chargesize:~/dev/pig-apache/pig/trunk]java -cp /tmp/svncheckout/trunk/pig.jar org.apache.pig.Main -x local -e "a = load 'a.txt'; b = load 'b.txt'; c = load 'c.txt'; d = join a by \$0, b by \$0 using \"replicated\"; dump d; e = join a by \$0, c by \$0 using \"replicated\"; dump e;"
2009-09-11 15:27:54,852 [main] INFO org.apache.pig.Main - Logging error messages to: /homes/pradeepk/dev/pig-apache/pig/trunk/pig_1252708074851.log
2009-09-11 15:27:55,217 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully stored result in: "file:/tmp/temp-1388892738/tmp1991974517"
2009-09-11 15:27:55,218 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records written : 2
2009-09-11 15:27:55,218 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written : 0
2009-09-11 15:27:55,218 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-09-11 15:27:55,218 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(1,2,3,1,x)
(3,4,5,3,a)
2009-09-11 15:27:55,253 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully stored result in: "file:/tmp/temp-1388892738/tmp84396309"
2009-09-11 15:27:55,253 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records written : 1
2009-09-11 15:27:55,253 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written : 0
2009-09-11 15:27:55,254 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
2009-09-11 15:27:55,254 [main] INFO org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
(1,2,3,1,20,30)
[pradeepk@chargesize:~/dev/pig-apache/pig/trunk]
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Fix For: 0.4.0
>
> Attachments: 641.patch, 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-641) Fragment replicate join does not work in
local mode
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-641:
-------------------------------
Attachment: 641.patch
I have added a local POFRJoinLocal operator that extends POFRJoin and uses the hashmap based implementation of joins similar to POFRJoin.
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch, 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-641) Fragment replicate join does not work in
local mode
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shubham Chopra updated PIG-641:
-------------------------------
Attachment: 641.patch
Added a small modification in query parser to treat FRJoin as regular joins since FRJoin - the algorithm, is redundant in the local mode. Also added test cases similar to the ones for FRJoin for the local mode as well.
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-641) Fragment replicate join does not work
in local mode
Posted by "Shubham Chopra (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672339#action_12672339 ]
Shubham Chopra commented on PIG-641:
------------------------------------
There is no single join operator in local mode. Like the MR mode, it depended on the rewriting of joins to cogroup + foreach done in the parser. But FRJoinLocal can be the single join operator now and can work as an alternative if anymore join improvements are added to the MR mode.
> Fragment replicate join does not work in local mode
> ---------------------------------------------------
>
> Key: PIG-641
> URL: https://issues.apache.org/jira/browse/PIG-641
> Project: Pig
> Issue Type: Bug
> Reporter: Olga Natkovich
> Assignee: Shubham Chopra
> Attachments: 641.patch, 641.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.