You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Mostafa Mokhtar (JIRA)" <ji...@apache.org> on 2017/06/20 23:05:00 UTC
[jira] [Created] (HADOOP-14558) RPC requests on a secure cluster
are 10x slower due to expensive encryption and decryption
Mostafa Mokhtar created HADOOP-14558:
----------------------------------------
Summary: RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
Key: HADOOP-14558
URL: https://issues.apache.org/jira/browse/HADOOP-14558
Project: Hadoop Common
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.6.0
Reporter: Mostafa Mokhtar
Priority: Critical
While running a performance tests for Impala comparing secure and un-secure clusters I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.
hadoop.rpc.protection is set to privacy
Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata loading.
The majority of the slowdown is coming from the two threads below.
{code}
Stack Trace Sample Count Percentage(%)
org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203 46.505
java.io.DataInputStream.readInt() 5,039 45.039
java.io.BufferedInputStream.read() 5,038 45.03
java.io.BufferedInputStream.fill() 5,038 45.03
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) 5,036 45.013
java.io.FilterInputStream.read(byte[], int, int) 5,036 45.013
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int, int) 5,036 45.013
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket() 5,035 45.004
com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775 42.68
sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int, MessageProp) 4,775 42.68
sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp) 4,768 42.617
sun.security.jgss.krb5.WrapToken.getData() 4,714 42.134
sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714 42.134
sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int, byte[], int) 3,083 27.556
sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int, byte[], int) 3,078 27.512
sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[], int, int) 3,076 27.494
sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[], byte[], int, int) 3,076 27.494
{code}
And
{code}
Stack Trace Sample Count Percentage(%)
java.lang.Thread.run() 3,379 30.202
java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 3,379 30.202
java.util.concurrent.FutureTask.run() 3,367 30.095
java.util.concurrent.Executors$RunnableAdapter.call() 3,367 30.095
org.apache.hadoop.ipc.Client$Connection$3.run() 3,367 30.095
java.io.DataOutputStream.flush() 3,367 30.095
java.io.BufferedOutputStream.flush() 3,367 30.095
java.io.BufferedOutputStream.flushBuffer() 3,367 30.095
org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int, int) 3,367 30.095
com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281 29.326
sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281 29.326
sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp) 3,280 29.317
sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[], int, int) 1,872 16.732
sun.security.jgss.krb5.WrapToken.encode() 1,407 12.576
{code}
This is the Impala Catalog thread which initiates the NameNode request
{code}
Stack Trace Sample Count Percentage(%)
org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest) 2,378 21.255
org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table) 2,378 21.255
org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table) 2,351 21.014
org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table, boolean, boolean, Set) 2,351 21.014
org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient, Set, boolean) 2,350 21.005
org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List) 2,326 20.79
org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor, HdfsPartition) 2,233 19.959
org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition) 1,998 17.858
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path) 1,496 13.371
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org