You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Jeff Zhang (JIRA)" <ji...@apache.org> on 2009/12/14 04:58:18 UTC
[jira] Created: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Move splitable logic from pig latin to InputFormat
--------------------------------------------------
Key: PIG-1148
URL: https://issues.apache.org/jira/browse/PIG-1148
Project: Pig
Issue Type: Sub-task
Reporter: Jeff Zhang
Assignee: Jeff Zhang
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791552#action_12791552 ]
Pradeep Kamath commented on PIG-1148:
-------------------------------------
As part of the new load-store-redesign work, we plan to remove "split by file" from pig latin language and users will need to write an inputformat which can achieve the same and associate the loader with that inputformat (like in your example). I am confused on what this jira will do since "split by file" will no longer remain in the language - can you explain a little on what the patch for this jira will do?
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791705#action_12791705 ]
Jeff Zhang commented on PIG-1148:
---------------------------------
Hi Pradeep, sorry for confusing you. I think I should say this jira is about remove "split by file" from pig latin. Since InputFormat can support it. And I found the branch load-store-redesign still support "split by file" in pig-latin now, so I create this jira to remove it, I think it is deprecated.
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1148:
----------------------------
Fix Version/s: 0.7.0
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.7.0
>
> Attachments: PIG-1148.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792359#action_12792359 ]
Jeff Zhang commented on PIG-1148:
---------------------------------
Pradeep, agree with you about removing isSplittable flag in LOLoad and POLoad.
I will continue work on this.
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791712#action_12791712 ]
Pradeep Kamath commented on PIG-1148:
-------------------------------------
I agree we need to remove it from pig latin - while removing it from the grammar also check if we can remove the isSplittable flag in LOLoad and POLoad.
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1148:
--------------------------------
Resolution: Fixed
Release Note: split by 'file' is not not allowed as part of the load statement to process input files in one map. To achieve this users will have to use an InputFormat in the loader which can return one split for the whole file.
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
+1, Thanks for the contribution Jeff - I have committed this patch on your behalf.
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Attachments: PIG-1148.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794085#action_12794085 ]
Hadoop QA commented on PIG-1148:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428829/PIG-1148.patch
against trunk revision 893373.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 27 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/154/console
This message is automatically generated.
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Attachments: PIG-1148.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793822#action_12793822 ]
Richard Ding commented on PIG-1148:
-----------------------------------
Hi Jeff, I'm in the process of cleaning the deprecated code in load-store branch. Please let me know when you expect to complete this jira? Thanks.
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-1148:
----------------------------
Status: Patch Available (was: Open)
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Attachments: PIG-1148.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790273#action_12790273 ]
Pradeep Kamath commented on PIG-1148:
-------------------------------------
Hi Jeff,
With the new load store redesign (http://wiki.apache.org/pig/LoadStoreRedesignProposal) wouldn't this be achieved implcitly since the splits used by pig will be the ones returned from the InputFormat associated with the Loader. The plan was to remove SPLIT by 'file' from the language since with the new load-store design it will not be possible to support this from pig - So there will be no splitable logic left with that approach - where you thinking of some other way to support split by file?
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Zhang updated PIG-1148:
----------------------------
Attachment: PIG-1148.patch
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Attachments: PIG-1148.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791396#action_12791396 ]
Jeff Zhang commented on PIG-1148:
---------------------------------
Pradeep, I do not quite understand your meaning.
I'd like to explain my idea again. Now users have to user "split by 'file' " in pig-latin to force hadoop do not split file into InputSplit. I don't think it's a good idea to put too many features on pig-latin. The principle of load store redesign is to integrate the features of hadoop into pig as much as possible but do not want to tie Pig Latin tightly to Hadoop.
So my suggestion is that, if pig do not want to split a file, he can provide a LoadFunc whose InputFormat control the splitable, this InputFormat extends FileInputFormat and override method isSplitable(FileSystem fs, Path filename) to control the splitable.
here's the code snippet illustrating my idea:
{code}
public class MyPigStorage extends PigStorage{
@Override
public InputFormat getInputFormat() {
return new MyInputFormat();
}
}
public class MyInputFormat extends TextInputFormat{
protected boolean isSplitable(FileSystem fs, Path filename) {
return false;
}
{code}
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789976#action_12789976 ]
Jeff Zhang commented on PIG-1148:
---------------------------------
Currently when user do not want to split file, he have to put the logic in pig latin. Just like the following pig script
{code}
raw = LOAD '/data/input' USING PigStorage() SPLIT BY 'file'
{code}
After pig support InputFormat, pig will accept any kinds of input, and hadoop have InputFormat which can control the split.
So I suggest to move splitable logic from pig latin to InputFormat
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1148) Move splitable logic from pig latin to
InputFormat
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1148.
---------------------------
> Move splitable logic from pig latin to InputFormat
> --------------------------------------------------
>
> Key: PIG-1148
> URL: https://issues.apache.org/jira/browse/PIG-1148
> Project: Pig
> Issue Type: Sub-task
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.7.0
>
> Attachments: PIG-1148.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.