You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2011/08/11 23:38:27 UTC
[jira] [Created] (HIVE-2370) Improve RCFileCat performance
significantly
Improve RCFileCat performance significantly
-------------------------------------------
Key: HIVE-2370
URL: https://issues.apache.org/jira/browse/HIVE-2370
Project: Hive
Issue Type: Improvement
Components: CLI
Affects Versions: 0.8.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong
Priority: Minor
The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083768#comment-13083768 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/
-----------------------------------------------------------
(Updated 2011-08-11 22:44:48.620762)
Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
Summary (updated)
-------
This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
Initial:
0.32 MB/s
Change System.out to use bigger buffer (not line buffered)
1.7MB/s
Unchecked Get:
1.75MB/s
Use StringBuilder to construct each row before writing output.
3.7MB/s
Streamline decoding:
4.16 MB/s
Use StringBuilder to buffer multiple lines:
5 MB/s
Tuning buffer sizes:
5.15 MB/s
I also added a --verbose mode which writes progress updates to stderr.
This addresses bug HIVE-2370.
https://issues.apache.org/jira/browse/HIVE-2370
Diffs
-----
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
Diff: https://reviews.apache.org/r/1474/diff
Testing
-------
Used diff to check output was same as old version of RCFileCat
Thanks,
Tim
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083903#comment-13083903 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/
-----------------------------------------------------------
(Updated 2011-08-12 03:56:39.298860)
Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
Changes
-------
Turned magic numbers into named constants, enable output buffering only after arguments processed.
Summary
-------
This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
Initial:
0.32 MB/s
Change System.out to use bigger buffer (not line buffered)
1.7MB/s
Unchecked Get:
1.75MB/s
Use StringBuilder to construct each row before writing output.
3.7MB/s
Streamline decoding:
4.16 MB/s
Use StringBuilder to buffer multiple lines:
5 MB/s
Tuning buffer sizes:
5.15 MB/s
I also added a --verbose mode which writes progress updates to stderr.
This addresses bug HIVE-2370.
https://issues.apache.org/jira/browse/HIVE-2370
Diffs (updated)
-----
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
Diff: https://reviews.apache.org/r/1474/diff
Testing
-------
Used diff to check output was same as old version of RCFileCat
Thanks,
Tim
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch, rcfilecat_2011-08-11b.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083765#comment-13083765 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/
-----------------------------------------------------------
Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
Summary
-------
This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
Initial:
0.32 MB/s
Change System.out to use bigger buffer (not line buffered)
1.7MB/s
Unchecked Get:
1.75MB/s
Use StringBuilder to construct each row before writing output.
3.7MB/s
Streamline decoding:
4.16 MB/s
Use StringBuilder to buffer multiple lines:
5 MB/s
Tuning buffer sizes:
5.15 MB/s
I also added a --verbose mode which writes progress updates to stderr.
This addresses bug HIVE-2370.
https://issues.apache.org/jira/browse/HIVE-2370
Diffs
-----
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
Diff: https://reviews.apache.org/r/1474/diff
Testing
-------
Used diff to check output was same as old version of RCFileCat
Thanks,
Tim
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Tim Armstrong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083766#comment-13083766 ]
Tim Armstrong commented on HIVE-2370:
-------------------------------------
Diff is available on reviewboard:
https://reviews.apache.org/r/1474/
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084320#comment-13084320 ]
Ning Zhang commented on HIVE-2370:
----------------------------------
+1. Will commit if tests pass.
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch, rcfilecat_2011-08-11b.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083792#comment-13083792 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/#review1412
-----------------------------------------------------------
Great job! Does this number indicate the read and write speed or just the read (including decompression) part?
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java
<https://reviews.apache.org/r/1474/#comment3266>
can you remove all these TABs?
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java
<https://reviews.apache.org/r/1474/#comment3267>
make 2048 a static constant variable.
- Ning
On 2011-08-11 22:44:48, Tim Armstrong wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1474/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-11 22:44:48)
bq.
bq.
bq. Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
bq.
bq. Initial:
bq. 0.32 MB/s
bq.
bq. Change System.out to use bigger buffer (not line buffered)
bq. 1.7MB/s
bq.
bq. Unchecked Get:
bq. 1.75MB/s
bq.
bq. Use StringBuilder to construct each row before writing output.
bq. 3.7MB/s
bq.
bq. Streamline decoding:
bq. 4.16 MB/s
bq.
bq. Use StringBuilder to buffer multiple lines:
bq. 5 MB/s
bq.
bq. Tuning buffer sizes:
bq. 5.15 MB/s
bq.
bq.
bq. I also added a --verbose mode which writes progress updates to stderr.
bq.
bq.
bq. This addresses bug HIVE-2370.
bq. https://issues.apache.org/jira/browse/HIVE-2370
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
bq.
bq. Diff: https://reviews.apache.org/r/1474/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Used diff to check output was same as old version of RCFileCat
bq.
bq.
bq. Thanks,
bq.
bq. Tim
bq.
bq.
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Tim Armstrong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated HIVE-2370:
--------------------------------
Attachment: rcfilecat_2011-08-11b.patch
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch, rcfilecat_2011-08-11b.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Ning Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Zhang updated HIVE-2370:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.8.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Tim!
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: rcfilecat_2011-08-11.patch, rcfilecat_2011-08-11b.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084781#comment-13084781 ]
Hudson commented on HIVE-2370:
------------------------------
Integrated in Hive-trunk-h0.21 #893 (See [https://builds.apache.org/job/Hive-trunk-h0.21/893/])
HIVE-2370. Improve RCFileCat performance significantly (Tim Armstrong via Ning Zhang)
nzhang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157323
Files :
* /hive/trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: rcfilecat_2011-08-11.patch, rcfilecat_2011-08-11b.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083870#comment-13083870 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/#review1414
-----------------------------------------------------------
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java
<https://reviews.apache.org/r/1474/#comment3271>
This should probably be done after we finish processing the command line options.
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java
<https://reviews.apache.org/r/1474/#comment3269>
"1024*1024" should be replaced with a static final variable.
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java
<https://reviews.apache.org/r/1474/#comment3270>
Another constant that should be converted to a static final.
- Carl
On 2011-08-12 00:22:11, Tim Armstrong wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1474/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-08-12 00:22:11)
bq.
bq.
bq. Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
bq.
bq. Initial:
bq. 0.32 MB/s
bq.
bq. Change System.out to use bigger buffer (not line buffered)
bq. 1.7MB/s
bq.
bq. Unchecked Get:
bq. 1.75MB/s
bq.
bq. Use StringBuilder to construct each row before writing output.
bq. 3.7MB/s
bq.
bq. Streamline decoding:
bq. 4.16 MB/s
bq.
bq. Use StringBuilder to buffer multiple lines:
bq. 5 MB/s
bq.
bq. Tuning buffer sizes:
bq. 5.15 MB/s
bq.
bq.
bq. I also added a --verbose mode which writes progress updates to stderr.
bq.
bq.
bq. This addresses bug HIVE-2370.
bq. https://issues.apache.org/jira/browse/HIVE-2370
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
bq.
bq. Diff: https://reviews.apache.org/r/1474/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Used diff to check output was same as old version of RCFileCat
bq.
bq.
bq. Thanks,
bq.
bq. Tim
bq.
bq.
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Tim Armstrong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated HIVE-2370:
--------------------------------
Attachment: rcfilecat_2011-08-11.patch
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Tim Armstrong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated HIVE-2370:
--------------------------------
Status: Patch Available (was: Open)
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "Tim Armstrong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083835#comment-13083835 ]
Tim Armstrong commented on HIVE-2370:
-------------------------------------
I'm not sure exactly what you mean about read and write speeds.
I tested it reading a file off a remote DFS instance, redirecting the output to a local file. The time spend writing the output is negligible.
The largest part of time is spent doing unicode conversions to get it into a Java CharBuffer, and then writing it to the console. Decompression and deserialisation also takes up a large part of CPU time.
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2370) Improve RCFileCat performance
significantly
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083839#comment-13083839 ]
jiraposter@reviews.apache.org commented on HIVE-2370:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1474/
-----------------------------------------------------------
(Updated 2011-08-12 00:22:11.295461)
Review request for hive, Yongqiang He, Ning Zhang, and namit jain.
Changes
-------
Stripped out whitespace at end of line of old version.
Summary
-------
This patch improves rcfilecat performance enormously: throughput increased from 0.32MB/s to 5.15MB/s on one benchmark: 16x. There were a number of improvements I made to get to this performance:
Initial:
0.32 MB/s
Change System.out to use bigger buffer (not line buffered)
1.7MB/s
Unchecked Get:
1.75MB/s
Use StringBuilder to construct each row before writing output.
3.7MB/s
Streamline decoding:
4.16 MB/s
Use StringBuilder to buffer multiple lines:
5 MB/s
Tuning buffer sizes:
5.15 MB/s
I also added a --verbose mode which writes progress updates to stderr.
This addresses bug HIVE-2370.
https://issues.apache.org/jira/browse/HIVE-2370
Diffs (updated)
-----
trunk/cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java 1156839
Diff: https://reviews.apache.org/r/1474/diff
Testing
-------
Used diff to check output was same as old version of RCFileCat
Thanks,
Tim
> Improve RCFileCat performance significantly
> -------------------------------------------
>
> Key: HIVE-2370
> URL: https://issues.apache.org/jira/browse/HIVE-2370
> Project: Hive
> Issue Type: Improvement
> Components: CLI
> Affects Versions: 0.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Minor
> Attachments: rcfilecat_2011-08-11.patch
>
>
> The rcfilecat utility is extraordinarily slow: the throughput can be < 0.5 MB/s of compressed RCFile. We can implement much faster version to enable faster export of data from Hive.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira