You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Xiao Chen (JIRA)" <ji...@apache.org> on 2017/08/16 17:48:00 UTC
[jira] [Commented] (HADOOP-14688) Intern strings in KeyVersion and
EncryptedKeyVersion
[ https://issues.apache.org/jira/browse/HADOOP-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129144#comment-16129144 ]
Xiao Chen commented on HADOOP-14688:
------------------------------------
Bumping up on this...
I understand the concern of the added overhead for a normal operation like getFileEncryptionInfo. But from internal runs I did not see this interning causing any visible impact on NN throughput.
On the other hand, heap is pretty ugly without this one during re-encryption. Attaching a report ran from [jxray|http://www.jxray.com/]. The most related section is:
{quote}
7. DUPLICATE STRINGS
Total strings: 2,570,432 Unique strings: 1,033,993 Duplicate values: 3,559 Overhead: 170,572K (8.4%)
Top duplicate strings:
Ovhd Num char[]s Num objs Value
103,775K (5.1%) 830205 830205 "mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf"
23,042K (1.1%) 184337 184337 "0OFmjElLqXgtjvWKkgfRoLpUj92dHrEaQCPeh3VDh8V"
8,668K (0.4%) 184937 184937 "EEK"
2,853K (0.1%) 12176 12176 "POST /kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek?eek_op=reencrypt HTTP/1.1"
2,473K (0.1%) 12177 12177 "/kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek?eek_op=reencrypt"
2,298K (0.1%) 13374 13374 "/kms/v1/keyversion/mGscEhbOphwD8GQkGxVfHnV4PVo3lhmpPWurw3vGsLf/_eek"
{quote}
> Intern strings in KeyVersion and EncryptedKeyVersion
> ----------------------------------------------------
>
> Key: HADOOP-14688
> URL: https://issues.apache.org/jira/browse/HADOOP-14688
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
> Reporter: Xiao Chen
> Assignee: Xiao Chen
> Attachments: GC root of the String.png, HADOOP-14688.01.patch, heapdump analysis.png, jxray.report
>
>
> This is inspired by [~misha@cloudera.com]'s work on HDFS-11383.
> The key names and key version names are usually the same for a bunch of {{KeyVersion}} and {{EncryptedKeyVersion}}. We should not create duplicate objects for them.
> This is more important to HDFS-10899, where we try to re-encrypt all files' EDEKs in a given EZ. Those EDEKs all has the same key name, and mostly using no more than a couple of key version names.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org