You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Richard Ding (JIRA)" <ji...@apache.org> on 2009/12/09 23:23:18 UTC
[jira] Created: (PIG-1141) Make streaming work with the new
load-store interfaces
Make streaming work with the new load-store interfaces
-------------------------------------------------------
Key: PIG-1141
URL: https://issues.apache.org/jira/browse/PIG-1141
Project: Pig
Issue Type: Sub-task
Reporter: Richard Ding
Assignee: Richard Ding
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1141:
------------------------------
Attachment: PIG-1141.patch
This patch made changes following Alan's comments.
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-1141:
----------------------------
Fix Version/s: 0.7.0
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788360#action_12788360 ]
Richard Ding commented on PIG-1141:
-----------------------------------
This is to track the implementation of the proposed changes to streaming in http://wiki.apache.org/pig/LoadStoreRedesignProposal.
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1141:
------------------------------
Attachment: PIG-1141.patch
This patch also removes BinaryStorage and StreamOptimizer from the source (in the branch).
Here is the result of locally run 'commit-patch':
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 32 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 433 release audit warnings (more than the trunk's current 430 warnings).
{code}
The release audit warnings are all 'html' related.
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Ding updated PIG-1141:
------------------------------
Attachment: PIG-1141.patch
This patch implemented the proposed changes to streaming with the following modification:
* PigToStream interface:
{code}
package org.apache.pig.impl.streaming;
public interface PigToStream {
/**
* Given a tuple, produce an array of bytes to be passed to the streaming
* executable.
*/
public byte[] serialize(Tuple t) throws IOException;
}
{code}
* StreamToPig interface:
{code}
package org.apache.pig.impl.streaming;
public interface StreamToPig {
/**
* Given a byte array from a streaming executable, produce a tuple.
*/
public Tuple deserialize(byte[] bytes) throws IOException;
/**
* This will be called on the front end during planning and not on the back
* end during execution.
*
* @return the {@link LoadCaster} associated with this object.
* @throws IOException if there is an exception during LoadCaster
*/
public LoadCaster getLoadCaster() throws IOException;
}
{code}
* The class that provides the default implementation of these interfaces:
{code}
org.apache.pig.impl.streaming.PigStreaming
{code}
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792720#action_12792720 ]
Richard Ding commented on PIG-1141:
-----------------------------------
bq. In DefaultInputHandler.close, why was the code that flushes and closes stdin removed? Same question for DefaultOutputHandler and stdout. It seems like we still need to flush and close these streams properly.
Becuase there is no 'stdin' or 'stdout' to flush and close :)
bq. Similar to the above, close was removed from FileOutputHandler (but not FileInputHandler).
I want to do the same for FileInputHandler, but findbugs doesn't allow it :(
bq. Both PigToStream and StreamToPig interfaces should have some javadoc comments for the interface explaining what they do and why.
I'll add javadoc for the interfaces.
bq. In StorageUtil.parseFieldDel, you call Integer.valueOf(String) for both \u and \x. For \x you should instead use Integer.valueOf(String, 16).
This is copied (refactored) from the current PigStorage code, do we want to change it?
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793354#action_12793354 ]
Alan Gates commented on PIG-1141:
---------------------------------
+1, changes look good.
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath resolved PIG-1141.
---------------------------------
Resolution: Fixed
Hadoop Flags: [Incompatible change, Reviewed]
Patch committed to load-store-redesign branch, thanks Richard!
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1141.
---------------------------
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Fix For: 0.7.0
>
> Attachments: PIG-1141.patch, PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1141) Make streaming work with the new
load-store interfaces
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792688#action_12792688 ]
Alan Gates commented on PIG-1141:
---------------------------------
In DefaultInputHandler.close, why was the code that flushes and closes stdin removed? Same question for DefaultOutputHandler and stdout. It seems like we still need to flush and close these streams properly.
Similar to the above, close was removed from FileOutputHandler (but not FileInputHandler).
Both PigToStream and StreamToPig interfaces should have some javadoc comments for the interface explaining what they do and why.
In StorageUtil.parseFieldDel, you call Integer.valueOf(String) for both \u and \x. For \x you should instead use Integer.valueOf(String, 16).
> Make streaming work with the new load-store interfaces
> -------------------------------------------------------
>
> Key: PIG-1141
> URL: https://issues.apache.org/jira/browse/PIG-1141
> Project: Pig
> Issue Type: Sub-task
> Reporter: Richard Ding
> Assignee: Richard Ding
> Attachments: PIG-1141.patch, PIG-1141.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.