You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Mathieu Poumeyrol (JIRA)" <ji...@apache.org> on 2008/04/10 11:44:05 UTC
[jira] Created: (PIG-201) BufferedPositionedInputStream is not
buffered
BufferedPositionedInputStream is not buffered
---------------------------------------------
Key: PIG-201
URL: https://issues.apache.org/jira/browse/PIG-201
Project: Pig
Issue Type: Bug
Reporter: Mathieu Poumeyrol
BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.
I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.
FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-201) BufferedPositionedInputStream is not
buffered
Posted by "Mathieu Poumeyrol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587558#action_12587558 ]
Mathieu Poumeyrol commented on PIG-201:
---------------------------------------
Ho ! The 8Kb buffer size is just plain guess. I honestly have no idea about the "right" value.
> BufferedPositionedInputStream is not buffered
> ---------------------------------------------
>
> Key: PIG-201
> URL: https://issues.apache.org/jira/browse/PIG-201
> Project: Pig
> Issue Type: Bug
> Reporter: Mathieu Poumeyrol
> Attachments: BufferedPositionedInputStream.patch
>
>
> BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.
> I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.
> FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-201) BufferedPositionedInputStream is not
buffered
Posted by "Mathieu Poumeyrol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mathieu Poumeyrol updated PIG-201:
----------------------------------
Attachment: BufferedPositionedInputStream.patch
> BufferedPositionedInputStream is not buffered
> ---------------------------------------------
>
> Key: PIG-201
> URL: https://issues.apache.org/jira/browse/PIG-201
> Project: Pig
> Issue Type: Bug
> Reporter: Mathieu Poumeyrol
> Attachments: BufferedPositionedInputStream.patch
>
>
> BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.
> I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.
> FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-201) BufferedPositionedInputStream is not
buffered
Posted by "Mathieu Poumeyrol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mathieu Poumeyrol updated PIG-201:
----------------------------------
Status: Patch Available (was: Open)
> BufferedPositionedInputStream is not buffered
> ---------------------------------------------
>
> Key: PIG-201
> URL: https://issues.apache.org/jira/browse/PIG-201
> Project: Pig
> Issue Type: Bug
> Reporter: Mathieu Poumeyrol
> Attachments: BufferedPositionedInputStream.patch
>
>
> BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.
> I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.
> FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-201) BufferedPositionedInputStream is not
buffered
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587658#action_12587658 ]
Benjamin Reed commented on PIG-201:
-----------------------------------
The InputStream we get from Hadoop DFS should be buffered, so we don't do extra buffering in BufferedPositionedInputStream again. This is important because the buffering needs to be done before the compression codecs so that the positioning works out properly. Doing it after, like this patch does, will cause premature detection of end of split.
Having said all that, there obviously is a performance gain to be had. Perhaps we need to figure out why the buffering done by Hadoop DFS InputStream isn't helping us. If we do need to buffer, it should go into PigSlice.init() to buffer fsis.
> BufferedPositionedInputStream is not buffered
> ---------------------------------------------
>
> Key: PIG-201
> URL: https://issues.apache.org/jira/browse/PIG-201
> Project: Pig
> Issue Type: Bug
> Reporter: Mathieu Poumeyrol
> Attachments: BufferedPositionedInputStream.patch
>
>
> BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.
> I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.
> FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-201) BufferedPositionedInputStream is not
buffered
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-201:
-------------------------------
Status: Open (was: Patch Available)
clearing the patch based on Ben's comments
> BufferedPositionedInputStream is not buffered
> ---------------------------------------------
>
> Key: PIG-201
> URL: https://issues.apache.org/jira/browse/PIG-201
> Project: Pig
> Issue Type: Bug
> Reporter: Mathieu Poumeyrol
> Attachments: BufferedPositionedInputStream.patch
>
>
> BufferedPositionedInputStream is actualy not buffered, leading (I guess) to constant round trip to dfs as byte are read one by one. I just wrapped the provided input stream in the constructor in a good old BufferedInputStream.
> I measured a 40% performance boost on a script that reads and writes 3.7GB in dfs through PigStorage on one node. I guess the impact may be greater on a real hdfs cluster with actual network roundtrips.
> FYI, the issue was found while profiling with Yourkit java profiler. Usefull toy...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.