You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/06/21 11:26:00 UTC
[jira] [Resolved] (HADOOP-14558) RPC requests on a secure cluster
are 10x slower due to expensive encryption and decryption
[ https://issues.apache.org/jira/browse/HADOOP-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-14558.
-------------------------------------
Resolution: Duplicate
Duplicate of HADOOP-10768.
Problem here is that the java encryption APIs allocate new byte buffers, get expensive fast.
If it were to be fixed, the obvious solution would be to have an option for native encryption and a private API which would encrypt either in-place or to a preallocated buffer, the latter being able to compensate for the fact that encrypted blocks often get their sizes rounded up. Talk to the intel developers and see if they have time to play with this?
> RPC requests on a secure cluster are 10x slower due to expensive encryption and decryption
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-14558
> URL: https://issues.apache.org/jira/browse/HADOOP-14558
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.0
> Reporter: Mostafa Mokhtar
> Priority: Critical
> Labels: impala, metadata, rpc
>
> While running a performance tests for Impala comparing secure and un-secure clusters I noticed that metadata loading operations are 10x slower on a cluster with Kerberos+SSL enabled.
> hadoop.rpc.protection is set to privacy
> Any recommendations on how this can be mitigated? 10x slowdown is a big hit for metadata loading.
> The majority of the slowdown is coming from the two threads below.
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203 46.505
> java.io.DataInputStream.readInt() 5,039 45.039
> java.io.BufferedInputStream.read() 5,038 45.03
> java.io.BufferedInputStream.fill() 5,038 45.03
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int) 5,036 45.013
> java.io.FilterInputStream.read(byte[], int, int) 5,036 45.013
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int, int) 5,036 45.013
> org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket() 5,035 45.004
> com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775 42.68
> sun.security.jgss.GSSContextImpl.unwrap(byte[], int, int, MessageProp) 4,775 42.68
> sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp) 4,768 42.617
> sun.security.jgss.krb5.WrapToken.getData() 4,714 42.134
> sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
> sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714 42.134
> sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int, byte[], int) 3,083 27.556
> sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int, byte[], int) 3,078 27.512
> sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[], int, int) 3,076 27.494
> sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[], byte[], int, int) 3,076 27.494
> {code}
> And
> {code}
> Stack Trace Sample Count Percentage(%)
> java.lang.Thread.run() 3,379 30.202
> java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 3,379 30.202
> java.util.concurrent.FutureTask.run() 3,367 30.095
> java.util.concurrent.Executors$RunnableAdapter.call() 3,367 30.095
> org.apache.hadoop.ipc.Client$Connection$3.run() 3,367 30.095
> java.io.DataOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flush() 3,367 30.095
> java.io.BufferedOutputStream.flushBuffer() 3,367 30.095
> org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int, int) 3,367 30.095
> com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281 29.326
> sun.security.jgss.GSSContextImpl.wrap(byte[], int, int, MessageProp) 3,281 29.326
> sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp) 3,280 29.317
> sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[], int, int) 1,872 16.732
> sun.security.jgss.krb5.WrapToken.encode() 1,407 12.576
> {code}
> This is the Impala Catalog thread which initiates the NameNode request
> {code}
> Stack Trace Sample Count Percentage(%)
> org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
> org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest) 2,378 21.255
> org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table) 2,378 21.255
> org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table) 2,351 21.014
> org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient, Table, boolean, boolean, Set) 2,351 21.014
> org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient, Set, boolean) 2,350 21.005
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List) 2,326 20.79
> org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor, HdfsPartition) 2,233 19.959
> org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition) 1,998 17.858
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path) 1,496 13.371
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org