You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/02/16 23:30:02 UTC
[jira] Commented: (HIVE-266) Improve SerDe performance by using
Text instead of String
[ https://issues.apache.org/jira/browse/HIVE-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12674074#action_12674074 ]
Zheng Shao commented on HIVE-266:
---------------------------------
HIVE-270 LazySimpleSerDe avoids String.split() but it still creates a String object per column per row.
Also we will need to write routines for LazySimpleSerDe to directly serialize (append) int/long/byte/short etc to UTF-8 buffer.
> Improve SerDe performance by using Text instead of String
> ---------------------------------------------------------
>
> Key: HIVE-266
> URL: https://issues.apache.org/jira/browse/HIVE-266
> Project: Hadoop Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Affects Versions: 0.2.0
> Reporter: Zheng Shao
> Priority: Critical
>
> A recent performance study showed that 2 places in Hive code has exhibited large cpu usage percentage:
> 1. String.getBytes() (UTF-8 encoding)
> 2. String.split()
> We should replace String with Text object to:
> 1. Avoid UTF-8 decoding and encoding
> 2. Reuse the Text object and avoid creating new objects for each column in each row like in String.split()
> This is expected to give a big (20%+) performance improvement to Hive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.