You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Michael Han (JIRA)" <ji...@apache.org> on 2017/06/21 00:00:40 UTC
[jira] [Commented] (HADOOP-14558) RPC requests on a secure cluster
are 10x slower due to expensive encryption and decryption
[ https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056696#comment-16056696 ]
Michael Han commented on HADOOP-14558:
--------------------------------------
This could be a known issue described in HADOOP-10768
> RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-14558
> URL: https://issues.apache.org/jira/browse/HADOOP-14558
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.0
> Reporter: Mostafa Mokhtar
> Priority: Critical
> Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure clusters I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.
> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata loading.
> The majority of the slowdown is coming from the two threads below.
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203 46.505
> java.io.DataInputStream.readInt() 5,039 45.039
> java.io.BufferedInputStream.read() 5,038 45.03
> java.io.BufferedInputStream.fill() 5,038 45.03
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) 5,036 45.013
> java.io.FilterInputStream.read(byte[], int, int) 5,036 45.013
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int, int) 5,036 45.013
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket() 5,035 45.004
> com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775 42.68
> sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int, MessageProp) 4,775 42.68
> sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp) 4,768 42.617
> sun.security.jgss.krb5.WrapToken.getData() 4,714 42.134
> sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
> sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714 42.134
> sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int, byte[], int) 3,083 27.556
> sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int, byte[], int) 3,078 27.512
> sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[], int, int) 3,076 27.494
> sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[], byte[], int, int) 3,076 27.494
> {code}
> And
> {code}
> Stack Trace Sample Count Percentage(%)
> java.lang.Thread.run() 3,379 30.202
> java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 3,379 30.202
> java.util.concurrent.FutureTask.run() 3,367 30.095
> java.util.concurrent.Executors$RunnableAdapter.call() 3,367 30.095
> org.apache.hadoop.ipc.Client$Connection$3.run() 3,367 30.095
> java.io.DataOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flushBuffer() 3,367 30.095
> org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int, int) 3,367 30.095
> com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281 29.326
> sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281 29.326
> sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp) 3,280 29.317
> sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[], int, int) 1,872 16.732
> sun.security.jgss.krb5.WrapToken.encode() 1,407 12.576
> {code}
> This is the Impala Catalog thread which initiates the NameNode request
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest) 2,378 21.255
> org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table) 2,378 21.255
> org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table) 2,351 21.014
> org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table, boolean, boolean, Set) 2,351 21.014
> org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient, Set, boolean) 2,350 21.005
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List) 2,326 20.79
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor, HdfsPartition) 2,233 19.959
> org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition) 1,998 17.858
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path) 1,496 13.371
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org