You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org> on 2011/08/12 00:44:29 UTC
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083765#comment-13083765 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/
-----------------------------------------------------------
Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
Summary
-------
This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
Initial:
0.32 MB/s
Change System.out to use bigger buffer (not line buffered)
1.7MB/s
Unchecked Get:
1.75MB/s
Use StringBuilder to construct each row before writing output.
3.7MB/s
Streamline decoding:
4.16 MB/s
Use StringBuilder to buffer multiple lines:
5 MB/s
Tuning buffer sizes:
5.15 MB/s
I also added a --verbose mode which writes progress updates to stderr.
This addresses bug HIVE-2370.
https://issues.apache.org/jira/browse/HIVE-2370
Diffs
-----
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
Diff: https://reviews.apache.org/r/1474/diff
Testing
-------
Used diff to check output was same as old version of RCFileCat
Thanks,
Tim
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira