You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by Guang Yang <ko...@uber.com> on 2018/09/04 06:46:12 UTC
WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
through WebHDFS in Knox, the download speed is just about 11M/s. However,
if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by David Villarreal <dv...@hortonworks.com>.
Interesting. Nice work. 2x improvement is great!
From: Kevin Risden <kr...@apache.org>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Wednesday, October 10, 2018 at 12:48 PM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
I tried disabling GCM ciphers based on the following information:
* https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8
* https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20
The results for the read were:
* knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
This is a little more than a 2x speedup. There is also information in the links above that there should be more performance improvements with JDK 9+.
For the write side slow down, I found an issue with how Knox is handing the streaming data on writes only. I am looking into fixing this to get the write performance for HDFS improved.
Kevin Risden
On Wed, Oct 10, 2018 at 1:20 PM David Villarreal <dv...@hortonworks.com>> wrote:
I believe Curl has an option of what cipher to use.. You may also be able to force it at the server jvm level using /jre/lib/security/java.security
From: Sandeep Moré <mo...@gmail.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 6:39 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
I think this would be a good test, worth a try, not sure how we can force a certain cipher to be used perhaps a permutation combination of
ssl.include.ciphers, ssl.exclude.ciphers.
Best,
Sandeep
On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <dv...@hortonworks.com>> wrote:
Hi Kevin,
In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers?
https://calomel.org/aesni_ssl_performance.html
Try forcing a less secure cipher to be used in your environment. Do you then see better numbers?
dav
From:
Kevin Risden
<kr...@apache.org>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 1:05 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
@David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked.
@Guang - NP just took some time to sit down and look at it.
Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty.
* https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
* https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
* https://webtide.com/conscrypting-native-ssl-for-jetty/
Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further.
Kevin Risden
On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com>> wrote:
Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL.
On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <dv...@hortonworks.com>> wrote:
I bring this up because HDFS encryption saw an increase in performance.
https://issues.apache.org/jira/browse/HDFS-6606
https://issues.apache.org/jira/browse/HADOOP-10768
Maybe Knox can make some enhancements in this area?
From: David Villarreal <dv...@hortonworks.com>>
Date: Tuesday, October 9, 2018 at 10:34 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Hi Kevin,
Now increase your CPU processing power and show me the numbers.
Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration?
libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
From:
Kevin Risden
<kr...@apache.org>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 10:26 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Writes look to have performance impact as well:
* directly to webhdfs - ~2.6 seconds
* knox no ssl - ~29 seconds
* knox ssl - ~49.6 seconds
Kevin Risden
On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org>> wrote:
If I run two downloads concurrently:
1,073,741,824 46.1MB/s in 22s
1,073,741,824 51.3MB/s in 22s
So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow.
Kevin Risden
On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org>> wrote:
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop.
Rough timings with wget for a 1GB random file:
* directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
* knox no ssl - 1,073,741,824 264MB/s in 3.6s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
There is a significant decrease with Knox SSL for some reason.
Kevin Risden
On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org>> wrote:
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.
On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org>> wrote:
So I found this in the Knox issues list in JIRA:
https://issues.apache.org/jira/browse/KNOX-1221
It sounds familiar in terms of a slowdown when going through Knox.
Kevin Risden
On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>> wrote:
Hmmm yea curl for a single file should do the handshake once.
What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer.
Kevin Risden
On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com>> wrote:
I'm just using curl to download a single large file. So I suspect SSL handshake just happens once?
On Tue, Sep 11, 2018 at 12:02 PM
Kevin Risden
<kr...@apache.org>> wrote:
What client are you using to connect Knox? Is this for a single file or a bunch of files?
The SSL handshake can be slow if the client doesn't keep the connection open.
Kevin Risden
On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com>> wrote:
Thanks Larry. But the only difference is this part in my gateway-site.xml.
<property>
<name>ssl.enabled</name>
<value>false</value>
<description>Indicates whether SSL is enabled.</description>
</property>
On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>> wrote:
I really don't think that kind of difference should be expected from merely SSL overhead.
I don't however have any metrics to contradict it either since I do not run Knox without SSL.
Given the above, I am struggling coming up with a meaningful response to this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL though.
On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com>> wrote:
Any idea guys?
On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com>> wrote:
Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL.
Without Knox, the download speed can reach to 400M/s if I call Namenode directly. And with disabling SSL, the speed can reach to ~400M/s as well through Knox. But with SSL, the speed drops significantly to ~40M/s. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective?
Thanks,
Guang
On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <dv...@hortonworks.com>> wrote:
Hi Guang,
Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker.
Thanks,
David
From: Sean Roberts <sr...@hortonworks.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, September 4, 2018 at 1:53 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by Sandeep Moré <mo...@gmail.com>.
Awesome, I had seen GCM suck big time in the past.
Great work !
On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden <kr...@apache.org> wrote:
> I tried disabling GCM ciphers based on the following information:
> * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8
> *
> https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20
>
> The results for the read were:
> * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s
> * knox ssl - 1,073,741,824 54.3MB/s in 20s
>
> This is a little more than a 2x speedup. There is also information in the
> links above that there should be more performance improvements with JDK 9+.
>
> For the write side slow down, I found an issue with how Knox is handing
> the streaming data on writes only. I am looking into fixing this to get the
> write performance for HDFS improved.
>
> Kevin Risden
>
>
> On Wed, Oct 10, 2018 at 1:20 PM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
>> I believe Curl has an option of what cipher to use.. You may also be
>> able to force it at the server jvm level using
>> /jre/lib/security/java.security
>>
>>
>>
>>
>>
>> *From: *Sandeep Moré <mo...@gmail.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, October 9, 2018 at 6:39 PM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> I think this would be a good test, worth a try, not sure how we can force
>> a certain cipher to be used perhaps a permutation combination of
>>
>> ssl.include.ciphers, ssl.exclude.ciphers.
>>
>>
>>
>> Best,
>>
>> Sandeep
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> Hi Kevin,
>>
>>
>>
>> In my humble opinion, this has to do with cpu processing encryption in
>> general based on which cipher being used. Couldn’t the same type of
>> principals/improvements (hdfs encryption improvements) be done here for
>> let’s say for AES cipher suites? If the main bottleneck here is CPU
>> couldn’t you enhance encryption though hardware acceleration and you may
>> see better performance numbers?
>>
>>
>>
>> https://calomel.org/aesni_ssl_performance.html
>>
>>
>>
>> Try forcing a less secure cipher to be used in your environment. Do you
>> then see better numbers?
>>
>>
>>
>> dav
>>
>>
>>
>>
>>
>>
>> *From:*
>> *Kevin Risden*
>>
>>
>>
>>
>> * <krisden@apache.org <kr...@apache.org>> Reply-To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Date: Tuesday, October 9, 2018 at 1:05 PM To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> @David - Not sure what you mean since this is SSL/TLS and not related to
>> RPC encryption like the two JIRAs that you linked.
>>
>> @Guang - NP just took some time to sit down and look at it.
>>
>>
>>
>> Some preliminary investigation shows this may be the JDK implementation
>> of TLS/SSL that is slowing down the read path. I need to dig into it
>> further but found a few references showing that Java slowness for TLS/SSL
>> affects Jetty.
>>
>> -
>> https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
>> -
>> https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
>> - https://webtide.com/conscrypting-native-ssl-for-jetty/
>>
>> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
>> enable conscrypting (
>> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
>> With that I was able to get read performance on par with non ssl and native
>> webhdfs. The write side of the equation still has some performance
>> differences that need to be looked at further.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks Kevin conducting such experiment! This is exactly what I saw
>> before. It doesn't look right the download speed is 10x slower when
>> enabling SSL.
>>
>>
>>
>> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> I bring this up because HDFS encryption saw an increase in performance.
>>
>> https://issues.apache.org/jira/browse/HDFS-6606
>>
>>
>>
>> https://issues.apache.org/jira/browse/HADOOP-10768
>>
>>
>>
>> Maybe Knox can make some enhancements in this area?
>>
>>
>>
>> *From: *David Villarreal <dv...@hortonworks.com>
>> *Date: *Tuesday, October 9, 2018 at 10:34 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Hi Kevin,
>>
>> Now increase your CPU processing power and show me the numbers.
>>
>>
>>
>> Do we support AES-NI optimization with extended CPU instruction set for
>> AES hardware acceleration?
>>
>> libcrypto.so library that supports hardware acceleration, such as
>> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that
>> does not support AES-NI.)
>>
>>
>>
>>
>>
>>
>> *From: *
>>
>> *Kevin Risden*
>>
>>
>>
>>
>>
>> *<krisden@apache.org <kr...@apache.org>> Reply-To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> Writes look to have performance impact as well:
>>
>> - directly to webhdfs - ~2.6 seconds
>> - knox no ssl - ~29 seconds
>> - knox ssl - ~49.6 seconds
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> If I run two downloads concurrently:
>>
>>
>>
>> 1,073,741,824 46.1MB/s in 22s
>>
>> 1,073,741,824 51.3MB/s in 22s
>>
>>
>>
>> So it isn't a limitation of the Knox gateway itself in total bandwidth
>> but a per connection limitation somehow.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> So I was able to reproduce a slowdown with SSL with a pseudo distributed
>> HDFS setup on a single node with Knox running on the same node. This was
>> setup in Virtualbox on my laptop.
>>
>>
>>
>> Rough timings with wget for a 1GB random file:
>>
>> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
>> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
>> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>>
>> There is a significant decrease with Knox SSL for some reason.
>>
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>>
>> SSL handshake will likely happen at least twice.
>>
>> Once for the request through Knox to the NN then the redirect from the NN
>> to the DN goes all the way back to the client.
>>
>> So they have to follow the redirect and do the handshake to the DN.
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> So I found this in the Knox issues list in JIRA:
>>
>>
>>
>> https://issues.apache.org/jira/browse/KNOX-1221
>>
>>
>>
>> It sounds familiar in terms of a slowdown when going through Knox.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> Hmmm yea curl for a single file should do the handshake once.
>>
>>
>>
>> What are the system performance statistics during the SSL vs non SSL
>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>> if using that. Otherwise watching top may be helpful. It would be help to
>> determine if the Knox is working harder during the SSL transfer.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>
>> I'm just using curl to download a single large file. So I suspect SSL
>> handshake just happens once?
>>
>>
>>
>> On Tue, Sep 11, 2018 at 12:02 PM
>>
>> Kevin Risden
>>
>> <kr...@apache.org> wrote:
>>
>> What client are you using to connect Knox? Is this for a single file or a
>> bunch of files?
>>
>>
>>
>> The SSL handshake can be slow if the client doesn't keep the connection
>> open.
>>
>> Kevin Risden
>>
>>
>>
>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>>
>>
>>
>> *<property>*
>>
>> * <name>ssl.enabled</name>*
>>
>> * <value>false</value>*
>>
>> * <description>Indicates whether SSL is enabled.</description>*
>>
>> *</property>*
>>
>>
>>
>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>>
>> I really don't think that kind of difference should be expected from
>> merely SSL overhead.
>>
>> I don't however have any metrics to contradict it either since I do not
>> run Knox without SSL.
>>
>>
>>
>> Given the above, I am struggling coming up with a meaningful response to
>> this. :(
>>
>> I don't think you should see a 10 fold increase in speed by disabling SSL
>> though.
>>
>>
>>
>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>
>> Any idea guys?
>>
>>
>>
>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks guys! The issue seems exactly what David pointed out, which is
>> because of encrypted over SSL.
>>
>>
>>
>> Without Knox, the download speed can reach to *400M/s* if I call
>> Namenode directly. And with disabling SSL, the speed can reach to
>> *~400M/s* as well through Knox. But with SSL, the speed drops
>> significantly to *~40M/s*. I know it's because of encrypted, but it does
>> surprised me with such a difference. Is it normal from your perspective?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>>
>>
>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> Hi Guang,
>>
>>
>>
>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>> you will most likely see a very significant boost in throughput. Some
>> people have used more powerful computers to make encryption quicker.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> David
>>
>>
>>
>> *From: *Sean Roberts <sr...@hortonworks.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Guang – This is somewhat to be expected.
>>
>>
>>
>> When you talk to WebHDFS directly, the client can distribute the request
>> across many data nodes. Also, you are getting data directly from the source.
>>
>> With Knox, all traffic goes through the single Knox host. Knox is
>> responsible for fetching from the datanodes and consolidating to send to
>> you. This means overhead as it’s acting as a middle man, and lower network
>> capacity since only 1 host is serving data to you.
>>
>>
>>
>> Also, if running on a cloud provider, the Knox host may be a smaller
>> instance size with lower network capacity.
>>
>> --
>>
>> Sean Roberts
>>
>>
>>
>> *From: *Guang Yang <ko...@uber.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, 4 September 2018 at 07:46
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *WebHDFS performance issue in Knox
>>
>>
>>
>> Hi,
>>
>>
>>
>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>> if we download directly from datanode, the speed is about 40M/s at least.
>>
>>
>>
>> Are you guys aware of this problem? Any suggestion?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>>
>>
>>
>>
>>
>>
>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
For reference I summarized from this thread and put the results
here: KNOX-1221
The write specific performance improvement is here: KNOX-1521
Kevin Risden
On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden <kr...@apache.org> wrote:
> I tried disabling GCM ciphers based on the following information:
> * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8
> *
> https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20
>
> The results for the read were:
> * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s
> * knox ssl - 1,073,741,824 54.3MB/s in 20s
>
> This is a little more than a 2x speedup. There is also information in the
> links above that there should be more performance improvements with JDK 9+.
>
> For the write side slow down, I found an issue with how Knox is handing
> the streaming data on writes only. I am looking into fixing this to get the
> write performance for HDFS improved.
>
> Kevin Risden
>
>
> On Wed, Oct 10, 2018 at 1:20 PM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
>> I believe Curl has an option of what cipher to use.. You may also be
>> able to force it at the server jvm level using
>> /jre/lib/security/java.security
>>
>>
>>
>>
>>
>> *From: *Sandeep Moré <mo...@gmail.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, October 9, 2018 at 6:39 PM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> I think this would be a good test, worth a try, not sure how we can force
>> a certain cipher to be used perhaps a permutation combination of
>>
>> ssl.include.ciphers, ssl.exclude.ciphers.
>>
>>
>>
>> Best,
>>
>> Sandeep
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> Hi Kevin,
>>
>>
>>
>> In my humble opinion, this has to do with cpu processing encryption in
>> general based on which cipher being used. Couldn’t the same type of
>> principals/improvements (hdfs encryption improvements) be done here for
>> let’s say for AES cipher suites? If the main bottleneck here is CPU
>> couldn’t you enhance encryption though hardware acceleration and you may
>> see better performance numbers?
>>
>>
>>
>> https://calomel.org/aesni_ssl_performance.html
>>
>>
>>
>> Try forcing a less secure cipher to be used in your environment. Do you
>> then see better numbers?
>>
>>
>>
>> dav
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:Kevin Risden <krisden@apache.org <kr...@apache.org>> Reply-To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Date: Tuesday, October 9, 2018 at 1:05 PM To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> @David - Not sure what you mean since this is SSL/TLS and not related to
>> RPC encryption like the two JIRAs that you linked.
>>
>> @Guang - NP just took some time to sit down and look at it.
>>
>>
>>
>> Some preliminary investigation shows this may be the JDK implementation
>> of TLS/SSL that is slowing down the read path. I need to dig into it
>> further but found a few references showing that Java slowness for TLS/SSL
>> affects Jetty.
>>
>> -
>> https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
>> -
>> https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
>> - https://webtide.com/conscrypting-native-ssl-for-jetty/
>>
>> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
>> enable conscrypting (
>> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
>> With that I was able to get read performance on par with non ssl and native
>> webhdfs. The write side of the equation still has some performance
>> differences that need to be looked at further.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks Kevin conducting such experiment! This is exactly what I saw
>> before. It doesn't look right the download speed is 10x slower when
>> enabling SSL.
>>
>>
>>
>> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> I bring this up because HDFS encryption saw an increase in performance.
>>
>> https://issues.apache.org/jira/browse/HDFS-6606
>>
>>
>>
>> https://issues.apache.org/jira/browse/HADOOP-10768
>>
>>
>>
>> Maybe Knox can make some enhancements in this area?
>>
>>
>>
>> *From: *David Villarreal <dv...@hortonworks.com>
>> *Date: *Tuesday, October 9, 2018 at 10:34 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Hi Kevin,
>>
>> Now increase your CPU processing power and show me the numbers.
>>
>>
>>
>> Do we support AES-NI optimization with extended CPU instruction set for
>> AES hardware acceleration?
>>
>> libcrypto.so library that supports hardware acceleration, such as
>> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that
>> does not support AES-NI.)
>>
>>
>>
>>
>>
>>
>> *From: *
>>
>> *Kevin Risden*
>>
>>
>>
>>
>>
>> *<krisden@apache.org <kr...@apache.org>> Reply-To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> Writes look to have performance impact as well:
>>
>> - directly to webhdfs - ~2.6 seconds
>> - knox no ssl - ~29 seconds
>> - knox ssl - ~49.6 seconds
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> If I run two downloads concurrently:
>>
>>
>>
>> 1,073,741,824 46.1MB/s in 22s
>>
>> 1,073,741,824 51.3MB/s in 22s
>>
>>
>>
>> So it isn't a limitation of the Knox gateway itself in total bandwidth
>> but a per connection limitation somehow.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> So I was able to reproduce a slowdown with SSL with a pseudo distributed
>> HDFS setup on a single node with Knox running on the same node. This was
>> setup in Virtualbox on my laptop.
>>
>>
>>
>> Rough timings with wget for a 1GB random file:
>>
>> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
>> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
>> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>>
>> There is a significant decrease with Knox SSL for some reason.
>>
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>>
>> SSL handshake will likely happen at least twice.
>>
>> Once for the request through Knox to the NN then the redirect from the NN
>> to the DN goes all the way back to the client.
>>
>> So they have to follow the redirect and do the handshake to the DN.
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> So I found this in the Knox issues list in JIRA:
>>
>>
>>
>> https://issues.apache.org/jira/browse/KNOX-1221
>>
>>
>>
>> It sounds familiar in terms of a slowdown when going through Knox.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> Hmmm yea curl for a single file should do the handshake once.
>>
>>
>>
>> What are the system performance statistics during the SSL vs non SSL
>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>> if using that. Otherwise watching top may be helpful. It would be help to
>> determine if the Knox is working harder during the SSL transfer.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>
>> I'm just using curl to download a single large file. So I suspect SSL
>> handshake just happens once?
>>
>>
>>
>> On Tue, Sep 11, 2018 at 12:02 PM
>>
>> Kevin Risden
>>
>> <kr...@apache.org> wrote:
>>
>> What client are you using to connect Knox? Is this for a single file or a
>> bunch of files?
>>
>>
>>
>> The SSL handshake can be slow if the client doesn't keep the connection
>> open.
>>
>> Kevin Risden
>>
>>
>>
>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>>
>>
>>
>> *<property>*
>>
>> * <name>ssl.enabled</name>*
>>
>> * <value>false</value>*
>>
>> * <description>Indicates whether SSL is enabled.</description>*
>>
>> *</property>*
>>
>>
>>
>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>>
>> I really don't think that kind of difference should be expected from
>> merely SSL overhead.
>>
>> I don't however have any metrics to contradict it either since I do not
>> run Knox without SSL.
>>
>>
>>
>> Given the above, I am struggling coming up with a meaningful response to
>> this. :(
>>
>> I don't think you should see a 10 fold increase in speed by disabling SSL
>> though.
>>
>>
>>
>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>
>> Any idea guys?
>>
>>
>>
>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks guys! The issue seems exactly what David pointed out, which is
>> because of encrypted over SSL.
>>
>>
>>
>> Without Knox, the download speed can reach to *400M/s* if I call
>> Namenode directly. And with disabling SSL, the speed can reach to
>> *~400M/s* as well through Knox. But with SSL, the speed drops
>> significantly to *~40M/s*. I know it's because of encrypted, but it does
>> surprised me with such a difference. Is it normal from your perspective?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>>
>>
>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> Hi Guang,
>>
>>
>>
>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>> you will most likely see a very significant boost in throughput. Some
>> people have used more powerful computers to make encryption quicker.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> David
>>
>>
>>
>> *From: *Sean Roberts <sr...@hortonworks.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Guang – This is somewhat to be expected.
>>
>>
>>
>> When you talk to WebHDFS directly, the client can distribute the request
>> across many data nodes. Also, you are getting data directly from the source.
>>
>> With Knox, all traffic goes through the single Knox host. Knox is
>> responsible for fetching from the datanodes and consolidating to send to
>> you. This means overhead as it’s acting as a middle man, and lower network
>> capacity since only 1 host is serving data to you.
>>
>>
>>
>> Also, if running on a cloud provider, the Knox host may be a smaller
>> instance size with lower network capacity.
>>
>> --
>>
>> Sean Roberts
>>
>>
>>
>> *From: *Guang Yang <ko...@uber.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, 4 September 2018 at 07:46
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *WebHDFS performance issue in Knox
>>
>>
>>
>> Hi,
>>
>>
>>
>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>> if we download directly from datanode, the speed is about 40M/s at least.
>>
>>
>>
>> Are you guys aware of this problem? Any suggestion?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>>
>>
>>
>>
>>
>>
>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
I tried disabling GCM ciphers based on the following information:
* https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8
*
https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20
The results for the read were:
* knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
This is a little more than a 2x speedup. There is also information in the
links above that there should be more performance improvements with JDK 9+.
For the write side slow down, I found an issue with how Knox is handing the
streaming data on writes only. I am looking into fixing this to get the
write performance for HDFS improved.
Kevin Risden
On Wed, Oct 10, 2018 at 1:20 PM David Villarreal <
dvillarreal@hortonworks.com> wrote:
> I believe Curl has an option of what cipher to use.. You may also be able
> to force it at the server jvm level using /jre/lib/security/java.security
>
>
>
>
>
> *From: *Sandeep Moré <mo...@gmail.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, October 9, 2018 at 6:39 PM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> I think this would be a good test, worth a try, not sure how we can force
> a certain cipher to be used perhaps a permutation combination of
>
> ssl.include.ciphers, ssl.exclude.ciphers.
>
>
>
> Best,
>
> Sandeep
>
>
>
>
>
> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
> Hi Kevin,
>
>
>
> In my humble opinion, this has to do with cpu processing encryption in
> general based on which cipher being used. Couldn’t the same type of
> principals/improvements (hdfs encryption improvements) be done here for
> let’s say for AES cipher suites? If the main bottleneck here is CPU
> couldn’t you enhance encryption though hardware acceleration and you may
> see better performance numbers?
>
>
>
> https://calomel.org/aesni_ssl_performance.html
>
>
>
> Try forcing a less secure cipher to be used in your environment. Do you
> then see better numbers?
>
>
>
> dav
>
>
>
>
>
>
>
>
>
>
> *From:Kevin Risden <krisden@apache.org <kr...@apache.org>> Reply-To:
> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
> <us...@knox.apache.org>> Date: Tuesday, October 9, 2018 at 1:05 PM To:
> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
> <us...@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox*
>
>
>
> @David - Not sure what you mean since this is SSL/TLS and not related to
> RPC encryption like the two JIRAs that you linked.
>
> @Guang - NP just took some time to sit down and look at it.
>
>
>
> Some preliminary investigation shows this may be the JDK implementation of
> TLS/SSL that is slowing down the read path. I need to dig into it further
> but found a few references showing that Java slowness for TLS/SSL affects
> Jetty.
>
> -
> https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
> -
> https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
> - https://webtide.com/conscrypting-native-ssl-for-jetty/
>
> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
> enable conscrypting (
> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
> With that I was able to get read performance on par with non ssl and native
> webhdfs. The write side of the equation still has some performance
> differences that need to be looked at further.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com> wrote:
>
> Thanks Kevin conducting such experiment! This is exactly what I saw
> before. It doesn't look right the download speed is 10x slower when
> enabling SSL.
>
>
>
> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal <dv...@hortonworks.com>
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
>
> *From: *
>
> *Kevin Risden*
>
>
>
>
>
> *<krisden@apache.org <kr...@apache.org>> Reply-To: "user@knox.apache.org
> <us...@knox.apache.org>" <user@knox.apache.org <us...@knox.apache.org>> Date:
> Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org
> <us...@knox.apache.org>" <user@knox.apache.org <us...@knox.apache.org>>
> Subject: Re: WebHDFS performance issue in Knox*
>
>
>
> Writes look to have performance impact as well:
>
> - directly to webhdfs - ~2.6 seconds
> - knox no ssl - ~29 seconds
> - knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s in 22s
>
> 1,073,741,824 51.3MB/s in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
>
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
>
> Kevin Risden
>
>
>
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>
> Hmmm yea curl for a single file should do the handshake once.
>
>
>
> What are the system performance statistics during the SSL vs non SSL
> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
> if using that. Otherwise watching top may be helpful. It would be help to
> determine if the Knox is working harder during the SSL transfer.
>
>
> Kevin Risden
>
>
>
>
>
> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>
> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
>
>
> On Tue, Sep 11, 2018 at 12:02 PM
>
> Kevin Risden
>
> <kr...@apache.org> wrote:
>
> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
>
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
>
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>
> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>
>
>
> *<property>*
>
> * <name>ssl.enabled</name>*
>
> * <value>false</value>*
>
> * <description>Indicates whether SSL is enabled.</description>*
>
> *</property>*
>
>
>
> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>
> I really don't think that kind of difference should be expected from
> merely SSL overhead.
>
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
>
>
> Given the above, I am struggling coming up with a meaningful response to
> this. :(
>
> I don't think you should see a 10 fold increase in speed by disabling SSL
> though.
>
>
>
> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>
> Any idea guys?
>
>
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>
> Thanks guys! The issue seems exactly what David pointed out, which is
> because of encrypted over SSL.
>
>
>
> Without Knox, the download speed can reach to *400M/s* if I call Namenode
> directly. And with disabling SSL, the speed can reach to *~400M/s* as
> well through Knox. But with SSL, the speed drops significantly to *~40M/s*.
> I know it's because of encrypted, but it does surprised me with such a
> difference. Is it normal from your perspective?
>
>
>
> Thanks,
>
> Guang
>
>
>
> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
> Hi Guang,
>
>
>
> Keep in mind the data is being encrypted over SSL. If you disable SSL you
> will most likely see a very significant boost in throughput. Some people
> have used more powerful computers to make encryption quicker.
>
>
>
> Thanks,
>
>
>
> David
>
>
>
> *From: *Sean Roberts <sr...@hortonworks.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, September 4, 2018 at 1:53 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Guang – This is somewhat to be expected.
>
>
>
> When you talk to WebHDFS directly, the client can distribute the request
> across many data nodes. Also, you are getting data directly from the source.
>
> With Knox, all traffic goes through the single Knox host. Knox is
> responsible for fetching from the datanodes and consolidating to send to
> you. This means overhead as it’s acting as a middle man, and lower network
> capacity since only 1 host is serving data to you.
>
>
>
> Also, if running on a cloud provider, the Knox host may be a smaller
> instance size with lower network capacity.
>
> --
>
> Sean Roberts
>
>
>
> *From: *Guang Yang <ko...@uber.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, 4 September 2018 at 07:46
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *WebHDFS performance issue in Knox
>
>
>
> Hi,
>
>
>
> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
> through WebHDFS in Knox, the download speed is just about 11M/s. However,
> if we download directly from datanode, the speed is about 40M/s at least.
>
>
>
> Are you guys aware of this problem? Any suggestion?
>
>
>
> Thanks,
>
> Guang
>
>
>
>
>
>
>
>
Re: WebHDFS performance issue in Knox
Posted by David Villarreal <dv...@hortonworks.com>.
I believe Curl has an option of what cipher to use.. You may also be able to force it at the server jvm level using /jre/lib/security/java.security
From: Sandeep Moré <mo...@gmail.com>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, October 9, 2018 at 6:39 PM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
I think this would be a good test, worth a try, not sure how we can force a certain cipher to be used perhaps a permutation combination of
ssl.include.ciphers, ssl.exclude.ciphers.
Best,
Sandeep
On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <dv...@hortonworks.com>> wrote:
Hi Kevin,
In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers?
https://calomel.org/aesni_ssl_performance.html
Try forcing a less secure cipher to be used in your environment. Do you then see better numbers?
dav
From: Kevin Risden <kr...@apache.org>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 1:05 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
@David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked.
@Guang - NP just took some time to sit down and look at it.
Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty.
* https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
* https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
* https://webtide.com/conscrypting-native-ssl-for-jetty/
Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further.
Kevin Risden
On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com>> wrote:
Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL.
On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <dv...@hortonworks.com>> wrote:
I bring this up because HDFS encryption saw an increase in performance.
https://issues.apache.org/jira/browse/HDFS-6606
https://issues.apache.org/jira/browse/HADOOP-10768
Maybe Knox can make some enhancements in this area?
From: David Villarreal <dv...@hortonworks.com>>
Date: Tuesday, October 9, 2018 at 10:34 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Hi Kevin,
Now increase your CPU processing power and show me the numbers.
Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration?
libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
From:
Kevin Risden
<kr...@apache.org>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 10:26 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Writes look to have performance impact as well:
* directly to webhdfs - ~2.6 seconds
* knox no ssl - ~29 seconds
* knox ssl - ~49.6 seconds
Kevin Risden
On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org>> wrote:
If I run two downloads concurrently:
1,073,741,824 46.1MB/s in 22s
1,073,741,824 51.3MB/s in 22s
So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow.
Kevin Risden
On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org>> wrote:
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop.
Rough timings with wget for a 1GB random file:
* directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
* knox no ssl - 1,073,741,824 264MB/s in 3.6s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
There is a significant decrease with Knox SSL for some reason.
Kevin Risden
On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org>> wrote:
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.
On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org>> wrote:
So I found this in the Knox issues list in JIRA:
https://issues.apache.org/jira/browse/KNOX-1221
It sounds familiar in terms of a slowdown when going through Knox.
Kevin Risden
On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>> wrote:
Hmmm yea curl for a single file should do the handshake once.
What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer.
Kevin Risden
On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com>> wrote:
I'm just using curl to download a single large file. So I suspect SSL handshake just happens once?
On Tue, Sep 11, 2018 at 12:02 PM
Kevin Risden
<kr...@apache.org>> wrote:
What client are you using to connect Knox? Is this for a single file or a bunch of files?
The SSL handshake can be slow if the client doesn't keep the connection open.
Kevin Risden
On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com>> wrote:
Thanks Larry. But the only difference is this part in my gateway-site.xml.
<property>
<name>ssl.enabled</name>
<value>false</value>
<description>Indicates whether SSL is enabled.</description>
</property>
On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>> wrote:
I really don't think that kind of difference should be expected from merely SSL overhead.
I don't however have any metrics to contradict it either since I do not run Knox without SSL.
Given the above, I am struggling coming up with a meaningful response to this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL though.
On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com>> wrote:
Any idea guys?
On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com>> wrote:
Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL.
Without Knox, the download speed can reach to 400M/s if I call Namenode directly. And with disabling SSL, the speed can reach to ~400M/s as well through Knox. But with SSL, the speed drops significantly to ~40M/s. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective?
Thanks,
Guang
On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <dv...@hortonworks.com>> wrote:
Hi Guang,
Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker.
Thanks,
David
From: Sean Roberts <sr...@hortonworks.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, September 4, 2018 at 1:53 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by Sandeep Moré <mo...@gmail.com>.
I think this would be a good test, worth a try, not sure how we can force a
certain cipher to be used perhaps a permutation combination of
ssl.include.ciphers, ssl.exclude.ciphers.
Best,
Sandeep
On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <dv...@hortonworks.com>
wrote:
> Hi Kevin,
>
>
>
> In my humble opinion, this has to do with cpu processing encryption in
> general based on which cipher being used. Couldn’t the same type of
> principals/improvements (hdfs encryption improvements) be done here for
> let’s say for AES cipher suites? If the main bottleneck here is CPU
> couldn’t you enhance encryption though hardware acceleration and you may
> see better performance numbers?
>
>
>
> https://calomel.org/aesni_ssl_performance.html
>
>
>
> Try forcing a less secure cipher to be used in your environment. Do you
> then see better numbers?
>
>
>
> dav
>
>
>
>
>
> *From: *Kevin Risden <kr...@apache.org>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, October 9, 2018 at 1:05 PM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> @David - Not sure what you mean since this is SSL/TLS and not related to
> RPC encryption like the two JIRAs that you linked.
>
> @Guang - NP just took some time to sit down and look at it.
>
>
>
> Some preliminary investigation shows this may be the JDK implementation of
> TLS/SSL that is slowing down the read path. I need to dig into it further
> but found a few references showing that Java slowness for TLS/SSL affects
> Jetty.
>
> -
> https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
> -
> https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
> - https://webtide.com/conscrypting-native-ssl-for-jetty/
>
> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
> enable conscrypting (
> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
> With that I was able to get read performance on par with non ssl and native
> webhdfs. The write side of the equation still has some performance
> differences that need to be looked at further.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com> wrote:
>
> Thanks Kevin conducting such experiment! This is exactly what I saw
> before. It doesn't look right the download speed is 10x slower when
> enabling SSL.
>
>
>
> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal <dv...@hortonworks.com>
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
>
> *From: *
>
> *Kevin Risden*
>
>
>
>
>
> *<krisden@apache.org <kr...@apache.org>> Reply-To: "user@knox.apache.org
> <us...@knox.apache.org>" <user@knox.apache.org <us...@knox.apache.org>> Date:
> Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org
> <us...@knox.apache.org>" <user@knox.apache.org <us...@knox.apache.org>>
> Subject: Re: WebHDFS performance issue in Knox*
>
>
>
> Writes look to have performance impact as well:
>
> - directly to webhdfs - ~2.6 seconds
> - knox no ssl - ~29 seconds
> - knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s in 22s
>
> 1,073,741,824 51.3MB/s in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
>
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
>
> Kevin Risden
>
>
>
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>
> Hmmm yea curl for a single file should do the handshake once.
>
>
>
> What are the system performance statistics during the SSL vs non SSL
> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
> if using that. Otherwise watching top may be helpful. It would be help to
> determine if the Knox is working harder during the SSL transfer.
>
>
> Kevin Risden
>
>
>
>
>
> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>
> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
>
>
> On Tue, Sep 11, 2018 at 12:02 PM
>
> Kevin Risden
>
> <kr...@apache.org> wrote:
>
> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
>
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
>
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>
> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>
>
>
> *<property>*
>
> * <name>ssl.enabled</name>*
>
> * <value>false</value>*
>
> * <description>Indicates whether SSL is enabled.</description>*
>
> *</property>*
>
>
>
> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>
> I really don't think that kind of difference should be expected from
> merely SSL overhead.
>
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
>
>
> Given the above, I am struggling coming up with a meaningful response to
> this. :(
>
> I don't think you should see a 10 fold increase in speed by disabling SSL
> though.
>
>
>
> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>
> Any idea guys?
>
>
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>
> Thanks guys! The issue seems exactly what David pointed out, which is
> because of encrypted over SSL.
>
>
>
> Without Knox, the download speed can reach to *400M/s* if I call Namenode
> directly. And with disabling SSL, the speed can reach to *~400M/s* as
> well through Knox. But with SSL, the speed drops significantly to *~40M/s*.
> I know it's because of encrypted, but it does surprised me with such a
> difference. Is it normal from your perspective?
>
>
>
> Thanks,
>
> Guang
>
>
>
> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
> Hi Guang,
>
>
>
> Keep in mind the data is being encrypted over SSL. If you disable SSL you
> will most likely see a very significant boost in throughput. Some people
> have used more powerful computers to make encryption quicker.
>
>
>
> Thanks,
>
>
>
> David
>
>
>
> *From: *Sean Roberts <sr...@hortonworks.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, September 4, 2018 at 1:53 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Guang – This is somewhat to be expected.
>
>
>
> When you talk to WebHDFS directly, the client can distribute the request
> across many data nodes. Also, you are getting data directly from the source.
>
> With Knox, all traffic goes through the single Knox host. Knox is
> responsible for fetching from the datanodes and consolidating to send to
> you. This means overhead as it’s acting as a middle man, and lower network
> capacity since only 1 host is serving data to you.
>
>
>
> Also, if running on a cloud provider, the Knox host may be a smaller
> instance size with lower network capacity.
>
> --
>
> Sean Roberts
>
>
>
> *From: *Guang Yang <ko...@uber.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, 4 September 2018 at 07:46
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *WebHDFS performance issue in Knox
>
>
>
> Hi,
>
>
>
> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
> through WebHDFS in Knox, the download speed is just about 11M/s. However,
> if we download directly from datanode, the speed is about 40M/s at least.
>
>
>
> Are you guys aware of this problem? Any suggestion?
>
>
>
> Thanks,
>
> Guang
>
>
>
>
>
>
>
>
Re: WebHDFS performance issue in Knox
Posted by David Villarreal <dv...@hortonworks.com>.
Hi Kevin,
In my humble opinion, this has to do with cpu processing encryption in general based on which cipher being used. Couldn’t the same type of principals/improvements (hdfs encryption improvements) be done here for let’s say for AES cipher suites? If the main bottleneck here is CPU couldn’t you enhance encryption though hardware acceleration and you may see better performance numbers?
https://calomel.org/aesni_ssl_performance.html
Try forcing a less secure cipher to be used in your environment. Do you then see better numbers?
dav
From: Kevin Risden <kr...@apache.org>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, October 9, 2018 at 1:05 PM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
@David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked.
@Guang - NP just took some time to sit down and look at it.
Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty.
* https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
* https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
* https://webtide.com/conscrypting-native-ssl-for-jetty/
Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting (https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further.
Kevin Risden
On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com>> wrote:
Thanks Kevin conducting such experiment! This is exactly what I saw before. It doesn't look right the download speed is 10x slower when enabling SSL.
On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <dv...@hortonworks.com>> wrote:
I bring this up because HDFS encryption saw an increase in performance.
https://issues.apache.org/jira/browse/HDFS-6606
https://issues.apache.org/jira/browse/HADOOP-10768
Maybe Knox can make some enhancements in this area?
From: David Villarreal <dv...@hortonworks.com>>
Date: Tuesday, October 9, 2018 at 10:34 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Hi Kevin,
Now increase your CPU processing power and show me the numbers.
Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration?
libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
From:
Kevin Risden
<kr...@apache.org>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, October 9, 2018 at 10:26 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Writes look to have performance impact as well:
* directly to webhdfs - ~2.6 seconds
* knox no ssl - ~29 seconds
* knox ssl - ~49.6 seconds
Kevin Risden
On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org>> wrote:
If I run two downloads concurrently:
1,073,741,824 46.1MB/s in 22s
1,073,741,824 51.3MB/s in 22s
So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow.
Kevin Risden
On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org>> wrote:
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop.
Rough timings with wget for a 1GB random file:
* directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
* knox no ssl - 1,073,741,824 264MB/s in 3.6s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
There is a significant decrease with Knox SSL for some reason.
Kevin Risden
On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org>> wrote:
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.
On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org>> wrote:
So I found this in the Knox issues list in JIRA:
https://issues.apache.org/jira/browse/KNOX-1221
It sounds familiar in terms of a slowdown when going through Knox.
Kevin Risden
On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>> wrote:
Hmmm yea curl for a single file should do the handshake once.
What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer.
Kevin Risden
On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com>> wrote:
I'm just using curl to download a single large file. So I suspect SSL handshake just happens once?
On Tue, Sep 11, 2018 at 12:02 PM
Kevin Risden
<kr...@apache.org>> wrote:
What client are you using to connect Knox? Is this for a single file or a bunch of files?
The SSL handshake can be slow if the client doesn't keep the connection open.
Kevin Risden
On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com>> wrote:
Thanks Larry. But the only difference is this part in my gateway-site.xml.
<property>
<name>ssl.enabled</name>
<value>false</value>
<description>Indicates whether SSL is enabled.</description>
</property>
On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>> wrote:
I really don't think that kind of difference should be expected from merely SSL overhead.
I don't however have any metrics to contradict it either since I do not run Knox without SSL.
Given the above, I am struggling coming up with a meaningful response to this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL though.
On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com>> wrote:
Any idea guys?
On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com>> wrote:
Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL.
Without Knox, the download speed can reach to 400M/s if I call Namenode directly. And with disabling SSL, the speed can reach to ~400M/s as well through Knox. But with SSL, the speed drops significantly to ~40M/s. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective?
Thanks,
Guang
On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <dv...@hortonworks.com>> wrote:
Hi Guang,
Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker.
Thanks,
David
From: Sean Roberts <sr...@hortonworks.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, September 4, 2018 at 1:53 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
@David - Not sure what you mean since this is SSL/TLS and not related to
RPC encryption like the two JIRAs that you linked.
@Guang - NP just took some time to sit down and look at it.
Some preliminary investigation shows this may be the JDK implementation of
TLS/SSL that is slowing down the read path. I need to dig into it further
but found a few references showing that Java slowness for TLS/SSL affects
Jetty.
- https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
-
https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
- https://webtide.com/conscrypting-native-ssl-for-jetty/
Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
enable conscrypting (
https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
With that I was able to get read performance on par with non ssl and native
webhdfs. The write side of the equation still has some performance
differences that need to be looked at further.
Kevin Risden
On Tue, Oct 9, 2018 at 2:01 PM Guang Yang <ko...@uber.com> wrote:
> Thanks Kevin conducting such experiment! This is exactly what I saw
> before. It doesn't look right the download speed is 10x slower when
> enabling SSL.
>
> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
>> I bring this up because HDFS encryption saw an increase in performance.
>>
>> https://issues.apache.org/jira/browse/HDFS-6606
>>
>>
>>
>> https://issues.apache.org/jira/browse/HADOOP-10768
>>
>>
>>
>> Maybe Knox can make some enhancements in this area?
>>
>>
>>
>> *From: *David Villarreal <dv...@hortonworks.com>
>> *Date: *Tuesday, October 9, 2018 at 10:34 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Hi Kevin,
>>
>> Now increase your CPU processing power and show me the numbers.
>>
>>
>>
>> Do we support AES-NI optimization with extended CPU instruction set for
>> AES hardware acceleration?
>>
>> libcrypto.so library that supports hardware acceleration, such as
>> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that
>> does not support AES-NI.)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:Kevin Risden <krisden@apache.org <kr...@apache.org>> Reply-To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Date: Tuesday, October 9, 2018 at 10:26 AM To:
>> "user@knox.apache.org <us...@knox.apache.org>" <user@knox.apache.org
>> <us...@knox.apache.org>> Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> Writes look to have performance impact as well:
>>
>> - directly to webhdfs - ~2.6 seconds
>> - knox no ssl - ~29 seconds
>> - knox ssl - ~49.6 seconds
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> If I run two downloads concurrently:
>>
>>
>>
>> 1,073,741,824 46.1MB/s in 22s
>>
>> 1,073,741,824 51.3MB/s in 22s
>>
>>
>>
>> So it isn't a limitation of the Knox gateway itself in total bandwidth
>> but a per connection limitation somehow.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> So I was able to reproduce a slowdown with SSL with a pseudo distributed
>> HDFS setup on a single node with Knox running on the same node. This was
>> setup in Virtualbox on my laptop.
>>
>>
>>
>> Rough timings with wget for a 1GB random file:
>>
>> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
>> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
>> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>>
>> There is a significant decrease with Knox SSL for some reason.
>>
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>>
>> SSL handshake will likely happen at least twice.
>>
>> Once for the request through Knox to the NN then the redirect from the NN
>> to the DN goes all the way back to the client.
>>
>> So they have to follow the redirect and do the handshake to the DN.
>>
>>
>>
>>
>>
>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> So I found this in the Knox issues list in JIRA:
>>
>>
>>
>> https://issues.apache.org/jira/browse/KNOX-1221
>>
>>
>>
>> It sounds familiar in terms of a slowdown when going through Knox.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>>
>> Hmmm yea curl for a single file should do the handshake once.
>>
>>
>>
>> What are the system performance statistics during the SSL vs non SSL
>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>> if using that. Otherwise watching top may be helpful. It would be help to
>> determine if the Knox is working harder during the SSL transfer.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>
>> I'm just using curl to download a single large file. So I suspect SSL
>> handshake just happens once?
>>
>>
>>
>> On Tue, Sep 11, 2018 at 12:02 PM
>>
>> Kevin Risden
>>
>> <kr...@apache.org> wrote:
>>
>> What client are you using to connect Knox? Is this for a single file or a
>> bunch of files?
>>
>>
>>
>> The SSL handshake can be slow if the client doesn't keep the connection
>> open.
>>
>> Kevin Risden
>>
>>
>>
>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>>
>>
>>
>> *<property>*
>>
>> * <name>ssl.enabled</name>*
>>
>> * <value>false</value>*
>>
>> * <description>Indicates whether SSL is enabled.</description>*
>>
>> *</property>*
>>
>>
>>
>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>>
>> I really don't think that kind of difference should be expected from
>> merely SSL overhead.
>>
>> I don't however have any metrics to contradict it either since I do not
>> run Knox without SSL.
>>
>>
>>
>> Given the above, I am struggling coming up with a meaningful response to
>> this. :(
>>
>> I don't think you should see a 10 fold increase in speed by disabling SSL
>> though.
>>
>>
>>
>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>
>> Any idea guys?
>>
>>
>>
>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>
>> Thanks guys! The issue seems exactly what David pointed out, which is
>> because of encrypted over SSL.
>>
>>
>>
>> Without Knox, the download speed can reach to *400M/s* if I call
>> Namenode directly. And with disabling SSL, the speed can reach to
>> *~400M/s* as well through Knox. But with SSL, the speed drops
>> significantly to *~40M/s*. I know it's because of encrypted, but it does
>> surprised me with such a difference. Is it normal from your perspective?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>>
>>
>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>> Hi Guang,
>>
>>
>>
>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>> you will most likely see a very significant boost in throughput. Some
>> people have used more powerful computers to make encryption quicker.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> David
>>
>>
>>
>> *From: *Sean Roberts <sr...@hortonworks.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Guang – This is somewhat to be expected.
>>
>>
>>
>> When you talk to WebHDFS directly, the client can distribute the request
>> across many data nodes. Also, you are getting data directly from the source.
>>
>> With Knox, all traffic goes through the single Knox host. Knox is
>> responsible for fetching from the datanodes and consolidating to send to
>> you. This means overhead as it’s acting as a middle man, and lower network
>> capacity since only 1 host is serving data to you.
>>
>>
>>
>> Also, if running on a cloud provider, the Knox host may be a smaller
>> instance size with lower network capacity.
>>
>> --
>>
>> Sean Roberts
>>
>>
>>
>> *From: *Guang Yang <ko...@uber.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, 4 September 2018 at 07:46
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *WebHDFS performance issue in Knox
>>
>>
>>
>> Hi,
>>
>>
>>
>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>> if we download directly from datanode, the speed is about 40M/s at least.
>>
>>
>>
>> Are you guys aware of this problem? Any suggestion?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>>
>>
>>
>>
>>
>>
>>
Re: WebHDFS performance issue in Knox
Posted by Guang Yang <ko...@uber.com>.
Thanks Kevin conducting such experiment! This is exactly what I saw before.
It doesn't look right the download speed is 10x slower when enabling SSL.
On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
dvillarreal@hortonworks.com> wrote:
> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal <dv...@hortonworks.com>
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
> *From: *Kevin Risden <kr...@apache.org>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, October 9, 2018 at 10:26 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Writes look to have performance impact as well:
>
> - directly to webhdfs - ~2.6 seconds
> - knox no ssl - ~29 seconds
> - knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s in 22s
>
> 1,073,741,824 51.3MB/s in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
>
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
>
> Kevin Risden
>
>
>
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>
> Hmmm yea curl for a single file should do the handshake once.
>
>
>
> What are the system performance statistics during the SSL vs non SSL
> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
> if using that. Otherwise watching top may be helpful. It would be help to
> determine if the Knox is working harder during the SSL transfer.
>
>
> Kevin Risden
>
>
>
>
>
> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>
> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
>
>
> On Tue, Sep 11, 2018 at 12:02 PM
>
> Kevin Risden
>
> <kr...@apache.org> wrote:
>
> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
>
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
>
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>
> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>
>
>
> *<property>*
>
> * <name>ssl.enabled</name>*
>
> * <value>false</value>*
>
> * <description>Indicates whether SSL is enabled.</description>*
>
> *</property>*
>
>
>
> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>
> I really don't think that kind of difference should be expected from
> merely SSL overhead.
>
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
>
>
> Given the above, I am struggling coming up with a meaningful response to
> this. :(
>
> I don't think you should see a 10 fold increase in speed by disabling SSL
> though.
>
>
>
> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>
> Any idea guys?
>
>
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>
> Thanks guys! The issue seems exactly what David pointed out, which is
> because of encrypted over SSL.
>
>
>
> Without Knox, the download speed can reach to *400M/s* if I call Namenode
> directly. And with disabling SSL, the speed can reach to *~400M/s* as
> well through Knox. But with SSL, the speed drops significantly to *~40M/s*.
> I know it's because of encrypted, but it does surprised me with such a
> difference. Is it normal from your perspective?
>
>
>
> Thanks,
>
> Guang
>
>
>
> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
> Hi Guang,
>
>
>
> Keep in mind the data is being encrypted over SSL. If you disable SSL you
> will most likely see a very significant boost in throughput. Some people
> have used more powerful computers to make encryption quicker.
>
>
>
> Thanks,
>
>
>
> David
>
>
>
> *From: *Sean Roberts <sr...@hortonworks.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, September 4, 2018 at 1:53 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Guang – This is somewhat to be expected.
>
>
>
> When you talk to WebHDFS directly, the client can distribute the request
> across many data nodes. Also, you are getting data directly from the source.
>
> With Knox, all traffic goes through the single Knox host. Knox is
> responsible for fetching from the datanodes and consolidating to send to
> you. This means overhead as it’s acting as a middle man, and lower network
> capacity since only 1 host is serving data to you.
>
>
>
> Also, if running on a cloud provider, the Knox host may be a smaller
> instance size with lower network capacity.
>
> --
>
> Sean Roberts
>
>
>
> *From: *Guang Yang <ko...@uber.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, 4 September 2018 at 07:46
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *WebHDFS performance issue in Knox
>
>
>
> Hi,
>
>
>
> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
> through WebHDFS in Knox, the download speed is just about 11M/s. However,
> if we download directly from datanode, the speed is about 40M/s at least.
>
>
>
> Are you guys aware of this problem? Any suggestion?
>
>
>
> Thanks,
>
> Guang
>
>
>
>
>
>
>
>
Re: WebHDFS performance issue in Knox
Posted by David Villarreal <dv...@hortonworks.com>.
I bring this up because HDFS encryption saw an increase in performance.
https://issues.apache.org/jira/browse/HDFS-6606
https://issues.apache.org/jira/browse/HADOOP-10768
Maybe Knox can make some enhancements in this area?
From: David Villarreal <dv...@hortonworks.com>
Date: Tuesday, October 9, 2018 at 10:34 AM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
Hi Kevin,
Now increase your CPU processing power and show me the numbers.
Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration?
libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
From: Kevin Risden <kr...@apache.org>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, October 9, 2018 at 10:26 AM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
Writes look to have performance impact as well:
* directly to webhdfs - ~2.6 seconds
* knox no ssl - ~29 seconds
* knox ssl - ~49.6 seconds
Kevin Risden
On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org>> wrote:
If I run two downloads concurrently:
1,073,741,824 46.1MB/s in 22s
1,073,741,824 51.3MB/s in 22s
So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow.
Kevin Risden
On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org>> wrote:
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop.
Rough timings with wget for a 1GB random file:
* directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
* knox no ssl - 1,073,741,824 264MB/s in 3.6s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
There is a significant decrease with Knox SSL for some reason.
Kevin Risden
On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org>> wrote:
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.
On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org>> wrote:
So I found this in the Knox issues list in JIRA:
https://issues.apache.org/jira/browse/KNOX-1221
It sounds familiar in terms of a slowdown when going through Knox.
Kevin Risden
On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>> wrote:
Hmmm yea curl for a single file should do the handshake once.
What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer.
Kevin Risden
On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com>> wrote:
I'm just using curl to download a single large file. So I suspect SSL handshake just happens once?
On Tue, Sep 11, 2018 at 12:02 PM
Kevin Risden
<kr...@apache.org>> wrote:
What client are you using to connect Knox? Is this for a single file or a bunch of files?
The SSL handshake can be slow if the client doesn't keep the connection open.
Kevin Risden
On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com>> wrote:
Thanks Larry. But the only difference is this part in my gateway-site.xml.
<property>
<name>ssl.enabled</name>
<value>false</value>
<description>Indicates whether SSL is enabled.</description>
</property>
On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>> wrote:
I really don't think that kind of difference should be expected from merely SSL overhead.
I don't however have any metrics to contradict it either since I do not run Knox without SSL.
Given the above, I am struggling coming up with a meaningful response to this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL though.
On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com>> wrote:
Any idea guys?
On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com>> wrote:
Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL.
Without Knox, the download speed can reach to 400M/s if I call Namenode directly. And with disabling SSL, the speed can reach to ~400M/s as well through Knox. But with SSL, the speed drops significantly to ~40M/s. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective?
Thanks,
Guang
On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <dv...@hortonworks.com>> wrote:
Hi Guang,
Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker.
Thanks,
David
From: Sean Roberts <sr...@hortonworks.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, September 4, 2018 at 1:53 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by David Villarreal <dv...@hortonworks.com>.
Hi Kevin,
Now increase your CPU processing power and show me the numbers.
Do we support AES-NI optimization with extended CPU instruction set for AES hardware acceleration?
libcrypto.so library that supports hardware acceleration, such as OpenSSL 1.0.1e. (Many OS versions have an older version of the library that does not support AES-NI.)
From: Kevin Risden <kr...@apache.org>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, October 9, 2018 at 10:26 AM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
Writes look to have performance impact as well:
* directly to webhdfs - ~2.6 seconds
* knox no ssl - ~29 seconds
* knox ssl - ~49.6 seconds
Kevin Risden
On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org>> wrote:
If I run two downloads concurrently:
1,073,741,824 46.1MB/s in 22s
1,073,741,824 51.3MB/s in 22s
So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow.
Kevin Risden
On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org>> wrote:
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop.
Rough timings with wget for a 1GB random file:
* directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
* knox no ssl - 1,073,741,824 264MB/s in 3.6s
* knox ssl - 1,073,741,824 54.3MB/s in 20s
There is a significant decrease with Knox SSL for some reason.
Kevin Risden
On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org>> wrote:
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.
On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org>> wrote:
So I found this in the Knox issues list in JIRA:
https://issues.apache.org/jira/browse/KNOX-1221
It sounds familiar in terms of a slowdown when going through Knox.
Kevin Risden
On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>> wrote:
Hmmm yea curl for a single file should do the handshake once.
What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer.
Kevin Risden
On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com>> wrote:
I'm just using curl to download a single large file. So I suspect SSL handshake just happens once?
On Tue, Sep 11, 2018 at 12:02 PM
Kevin Risden
<kr...@apache.org>> wrote:
What client are you using to connect Knox? Is this for a single file or a bunch of files?
The SSL handshake can be slow if the client doesn't keep the connection open.
Kevin Risden
On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com>> wrote:
Thanks Larry. But the only difference is this part in my gateway-site.xml.
<property>
<name>ssl.enabled</name>
<value>false</value>
<description>Indicates whether SSL is enabled.</description>
</property>
On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>> wrote:
I really don't think that kind of difference should be expected from merely SSL overhead.
I don't however have any metrics to contradict it either since I do not run Knox without SSL.
Given the above, I am struggling coming up with a meaningful response to this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL though.
On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com>> wrote:
Any idea guys?
On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com>> wrote:
Thanks guys! The issue seems exactly what David pointed out, which is because of encrypted over SSL.
Without Knox, the download speed can reach to 400M/s if I call Namenode directly. And with disabling SSL, the speed can reach to ~400M/s as well through Knox. But with SSL, the speed drops significantly to ~40M/s. I know it's because of encrypted, but it does surprised me with such a difference. Is it normal from your perspective?
Thanks,
Guang
On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <dv...@hortonworks.com>> wrote:
Hi Guang,
Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker.
Thanks,
David
From: Sean Roberts <sr...@hortonworks.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, September 4, 2018 at 1:53 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
Writes look to have performance impact as well:
- directly to webhdfs - ~2.6 seconds
- knox no ssl - ~29 seconds
- knox ssl - ~49.6 seconds
Kevin Risden
On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden <kr...@apache.org> wrote:
> If I run two downloads concurrently:
>
> 1,073,741,824 46.1MB/s in 22s
> 1,073,741,824 51.3MB/s in 22s
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
> Kevin Risden
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
>
>> So I was able to reproduce a slowdown with SSL with a pseudo distributed
>> HDFS setup on a single node with Knox running on the same node. This was
>> setup in Virtualbox on my laptop.
>>
>> Rough timings with wget for a 1GB random file:
>>
>> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
>> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
>> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>>
>> There is a significant decrease with Knox SSL for some reason.
>>
>> Kevin Risden
>>
>>
>> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>>
>>> SSL handshake will likely happen at least twice.
>>> Once for the request through Knox to the NN then the redirect from the
>>> NN to the DN goes all the way back to the client.
>>> So they have to follow the redirect and do the handshake to the DN.
>>>
>>>
>>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>>>
>>>> So I found this in the Knox issues list in JIRA:
>>>>
>>>> https://issues.apache.org/jira/browse/KNOX-1221
>>>>
>>>> It sounds familiar in terms of a slowdown when going through Knox.
>>>>
>>>> Kevin Risden
>>>>
>>>>
>>>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>
>>>> wrote:
>>>>
>>>>> Hmmm yea curl for a single file should do the handshake once.
>>>>>
>>>>> What are the system performance statistics during the SSL vs non SSL
>>>>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>>>>> if using that. Otherwise watching top may be helpful. It would be help to
>>>>> determine if the Knox is working harder during the SSL transfer.
>>>>>
>>>>> Kevin Risden
>>>>>
>>>>>
>>>>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>>>>
>>>>>> I'm just using curl to download a single large file. So I suspect SSL
>>>>>> handshake just happens once?
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 12:02 PM
>>>>>> Kevin Risden
>>>>>> <kr...@apache.org> wrote:
>>>>>>
>>>>>>> What client are you using to connect Knox? Is this for a single file
>>>>>>> or a bunch of files?
>>>>>>>
>>>>>>> The SSL handshake can be slow if the client doesn't keep the
>>>>>>> connection open.
>>>>>>>
>>>>>>> Kevin Risden
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>>>>>>
>>>>>>>> Thanks Larry. But the only difference is this part in my
>>>>>>>> gateway-site.xml.
>>>>>>>>
>>>>>>>> *<property>*
>>>>>>>> * <name>ssl.enabled</name>*
>>>>>>>> * <value>false</value>*
>>>>>>>> * <description>Indicates whether SSL is
>>>>>>>> enabled.</description>*
>>>>>>>> *</property>*
>>>>>>>>
>>>>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I really don't think that kind of difference should be expected
>>>>>>>>> from merely SSL overhead.
>>>>>>>>> I don't however have any metrics to contradict it either since I
>>>>>>>>> do not run Knox without SSL.
>>>>>>>>>
>>>>>>>>> Given the above, I am struggling coming up with a meaningful
>>>>>>>>> response to this. :(
>>>>>>>>> I don't think you should see a 10 fold increase in speed by
>>>>>>>>> disabling SSL though.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>>>>>>>
>>>>>>>>>> Any idea guys?
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks guys! The issue seems exactly what David pointed out,
>>>>>>>>>>> which is because of encrypted over SSL.
>>>>>>>>>>>
>>>>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I
>>>>>>>>>>> call Namenode directly. And with disabling SSL, the speed can reach to
>>>>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted,
>>>>>>>>>>> but it does surprised me with such a difference. Is it normal from your
>>>>>>>>>>> perspective?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Guang
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>>>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Guang,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Keep in mind the data is being encrypted over SSL. If you
>>>>>>>>>>>> disable SSL you will most likely see a very significant boost in
>>>>>>>>>>>> throughput. Some people have used more powerful computers to make
>>>>>>>>>>>> encryption quicker.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> David
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Guang – This is somewhat to be expected.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> When you talk to WebHDFS directly, the client can distribute
>>>>>>>>>>>> the request across many data nodes. Also, you are getting data directly
>>>>>>>>>>>> from the source.
>>>>>>>>>>>>
>>>>>>>>>>>> With Knox, all traffic goes through the single Knox host. Knox
>>>>>>>>>>>> is responsible for fetching from the datanodes and consolidating to send to
>>>>>>>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>>>>>>>> capacity since only 1 host is serving data to you.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Also, if running on a cloud provider, the Knox host may be a
>>>>>>>>>>>> smaller instance size with lower network capacity.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Sean Roberts
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download
>>>>>>>>>>>> a file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>>>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>>>>>>>> least.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Guang
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
If I run two downloads concurrently:
1,073,741,824 46.1MB/s in 22s
1,073,741,824 51.3MB/s in 22s
So it isn't a limitation of the Knox gateway itself in total bandwidth but
a per connection limitation somehow.
Kevin Risden
On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden <kr...@apache.org> wrote:
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
> Rough timings with wget for a 1GB random file:
>
> - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
> - knox no ssl - 1,073,741,824 264MB/s in 3.6s
> - knox ssl - 1,073,741,824 54.3MB/s in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
> Kevin Risden
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
>
>> SSL handshake will likely happen at least twice.
>> Once for the request through Knox to the NN then the redirect from the NN
>> to the DN goes all the way back to the client.
>> So they have to follow the redirect and do the handshake to the DN.
>>
>>
>> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>>
>>> So I found this in the Knox issues list in JIRA:
>>>
>>> https://issues.apache.org/jira/browse/KNOX-1221
>>>
>>> It sounds familiar in terms of a slowdown when going through Knox.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org>
>>> wrote:
>>>
>>>> Hmmm yea curl for a single file should do the handshake once.
>>>>
>>>> What are the system performance statistics during the SSL vs non SSL
>>>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>>>> if using that. Otherwise watching top may be helpful. It would be help to
>>>> determine if the Knox is working harder during the SSL transfer.
>>>>
>>>> Kevin Risden
>>>>
>>>>
>>>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>>>
>>>>> I'm just using curl to download a single large file. So I suspect SSL
>>>>> handshake just happens once?
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:02 PM
>>>>> Kevin Risden
>>>>> <kr...@apache.org> wrote:
>>>>>
>>>>>> What client are you using to connect Knox? Is this for a single file
>>>>>> or a bunch of files?
>>>>>>
>>>>>> The SSL handshake can be slow if the client doesn't keep the
>>>>>> connection open.
>>>>>>
>>>>>> Kevin Risden
>>>>>>
>>>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>>>>>
>>>>>>> Thanks Larry. But the only difference is this part in my
>>>>>>> gateway-site.xml.
>>>>>>>
>>>>>>> *<property>*
>>>>>>> * <name>ssl.enabled</name>*
>>>>>>> * <value>false</value>*
>>>>>>> * <description>Indicates whether SSL is
>>>>>>> enabled.</description>*
>>>>>>> *</property>*
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I really don't think that kind of difference should be expected
>>>>>>>> from merely SSL overhead.
>>>>>>>> I don't however have any metrics to contradict it either since I do
>>>>>>>> not run Knox without SSL.
>>>>>>>>
>>>>>>>> Given the above, I am struggling coming up with a meaningful
>>>>>>>> response to this. :(
>>>>>>>> I don't think you should see a 10 fold increase in speed by
>>>>>>>> disabling SSL though.
>>>>>>>>
>>>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>>>>>>
>>>>>>>>> Any idea guys?
>>>>>>>>>
>>>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks guys! The issue seems exactly what David pointed out,
>>>>>>>>>> which is because of encrypted over SSL.
>>>>>>>>>>
>>>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but
>>>>>>>>>> it does surprised me with such a difference. Is it normal from your
>>>>>>>>>> perspective?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Guang
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Guang,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Keep in mind the data is being encrypted over SSL. If you
>>>>>>>>>>> disable SSL you will most likely see a very significant boost in
>>>>>>>>>>> throughput. Some people have used more powerful computers to make
>>>>>>>>>>> encryption quicker.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Guang – This is somewhat to be expected.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>>>>>>> the source.
>>>>>>>>>>>
>>>>>>>>>>> With Knox, all traffic goes through the single Knox host. Knox
>>>>>>>>>>> is responsible for fetching from the datanodes and consolidating to send to
>>>>>>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>>>>>>> capacity since only 1 host is serving data to you.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Also, if running on a cloud provider, the Knox host may be a
>>>>>>>>>>> smaller instance size with lower network capacity.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Sean Roberts
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download
>>>>>>>>>>> a file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>>>>>>> least.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Guang
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
So I was able to reproduce a slowdown with SSL with a pseudo distributed
HDFS setup on a single node with Knox running on the same node. This was
setup in Virtualbox on my laptop.
Rough timings with wget for a 1GB random file:
- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s
- knox no ssl - 1,073,741,824 264MB/s in 3.6s
- knox ssl - 1,073,741,824 54.3MB/s in 20s
There is a significant decrease with Knox SSL for some reason.
Kevin Risden
On Sun, Sep 23, 2018 at 8:53 PM larry mccay <lm...@apache.org> wrote:
> SSL handshake will likely happen at least twice.
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
> So they have to follow the redirect and do the handshake to the DN.
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
>
>> So I found this in the Knox issues list in JIRA:
>>
>> https://issues.apache.org/jira/browse/KNOX-1221
>>
>> It sounds familiar in terms of a slowdown when going through Knox.
>>
>> Kevin Risden
>>
>>
>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>>
>>> Hmmm yea curl for a single file should do the handshake once.
>>>
>>> What are the system performance statistics during the SSL vs non SSL
>>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>>> if using that. Otherwise watching top may be helpful. It would be help to
>>> determine if the Knox is working harder during the SSL transfer.
>>>
>>> Kevin Risden
>>>
>>>
>>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>>
>>>> I'm just using curl to download a single large file. So I suspect SSL
>>>> handshake just happens once?
>>>>
>>>> On Tue, Sep 11, 2018 at 12:02 PM
>>>> Kevin Risden
>>>> <kr...@apache.org> wrote:
>>>>
>>>>> What client are you using to connect Knox? Is this for a single file
>>>>> or a bunch of files?
>>>>>
>>>>> The SSL handshake can be slow if the client doesn't keep the
>>>>> connection open.
>>>>>
>>>>> Kevin Risden
>>>>>
>>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>>>>
>>>>>> Thanks Larry. But the only difference is this part in my
>>>>>> gateway-site.xml.
>>>>>>
>>>>>> *<property>*
>>>>>> * <name>ssl.enabled</name>*
>>>>>> * <value>false</value>*
>>>>>> * <description>Indicates whether SSL is enabled.</description>*
>>>>>> *</property>*
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I really don't think that kind of difference should be expected from
>>>>>>> merely SSL overhead.
>>>>>>> I don't however have any metrics to contradict it either since I do
>>>>>>> not run Knox without SSL.
>>>>>>>
>>>>>>> Given the above, I am struggling coming up with a meaningful
>>>>>>> response to this. :(
>>>>>>> I don't think you should see a 10 fold increase in speed by
>>>>>>> disabling SSL though.
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>>>>>
>>>>>>>> Any idea guys?
>>>>>>>>
>>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which
>>>>>>>>> is because of encrypted over SSL.
>>>>>>>>>
>>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but
>>>>>>>>> it does surprised me with such a difference. Is it normal from your
>>>>>>>>> perspective?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Guang
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Guang,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Keep in mind the data is being encrypted over SSL. If you
>>>>>>>>>> disable SSL you will most likely see a very significant boost in
>>>>>>>>>> throughput. Some people have used more powerful computers to make
>>>>>>>>>> encryption quicker.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> David
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Guang – This is somewhat to be expected.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>>>>>> the source.
>>>>>>>>>>
>>>>>>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>>>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>>>>>> capacity since only 1 host is serving data to you.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Also, if running on a cloud provider, the Knox host may be a
>>>>>>>>>> smaller instance size with lower network capacity.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Sean Roberts
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a
>>>>>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>>>>>> least.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Guang
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
Re: WebHDFS performance issue in Knox
Posted by larry mccay <lm...@apache.org>.
SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN
to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.
On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden <kr...@apache.org> wrote:
> So I found this in the Knox issues list in JIRA:
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
> Kevin Risden
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
>
>> Hmmm yea curl for a single file should do the handshake once.
>>
>> What are the system performance statistics during the SSL vs non SSL
>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>> if using that. Otherwise watching top may be helpful. It would be help to
>> determine if the Knox is working harder during the SSL transfer.
>>
>> Kevin Risden
>>
>>
>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>>
>>> I'm just using curl to download a single large file. So I suspect SSL
>>> handshake just happens once?
>>>
>>> On Tue, Sep 11, 2018 at 12:02 PM
>>> Kevin Risden
>>> <kr...@apache.org> wrote:
>>>
>>>> What client are you using to connect Knox? Is this for a single file or
>>>> a bunch of files?
>>>>
>>>> The SSL handshake can be slow if the client doesn't keep the connection
>>>> open.
>>>>
>>>> Kevin Risden
>>>>
>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>>>
>>>>> Thanks Larry. But the only difference is this part in my
>>>>> gateway-site.xml.
>>>>>
>>>>> *<property>*
>>>>> * <name>ssl.enabled</name>*
>>>>> * <value>false</value>*
>>>>> * <description>Indicates whether SSL is enabled.</description>*
>>>>> *</property>*
>>>>>
>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I really don't think that kind of difference should be expected from
>>>>>> merely SSL overhead.
>>>>>> I don't however have any metrics to contradict it either since I do
>>>>>> not run Knox without SSL.
>>>>>>
>>>>>> Given the above, I am struggling coming up with a meaningful response
>>>>>> to this. :(
>>>>>> I don't think you should see a 10 fold increase in speed by disabling
>>>>>> SSL though.
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>>>>
>>>>>>> Any idea guys?
>>>>>>>
>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>>>>>
>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which
>>>>>>>> is because of encrypted over SSL.
>>>>>>>>
>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but
>>>>>>>> it does surprised me with such a difference. Is it normal from your
>>>>>>>> perspective?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Guang
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Guang,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Keep in mind the data is being encrypted over SSL. If you disable
>>>>>>>>> SSL you will most likely see a very significant boost in throughput. Some
>>>>>>>>> people have used more powerful computers to make encryption quicker.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> David
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Guang – This is somewhat to be expected.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>>>>> the source.
>>>>>>>>>
>>>>>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>>>>> capacity since only 1 host is serving data to you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also, if running on a cloud provider, the Knox host may be a
>>>>>>>>> smaller instance size with lower network capacity.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Sean Roberts
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a
>>>>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>>>>> least.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Guang
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
So I found this in the Knox issues list in JIRA:
https://issues.apache.org/jira/browse/KNOX-1221
It sounds familiar in terms of a slowdown when going through Knox.
Kevin Risden
On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden <kr...@apache.org> wrote:
> Hmmm yea curl for a single file should do the handshake once.
>
> What are the system performance statistics during the SSL vs non SSL
> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
> if using that. Otherwise watching top may be helpful. It would be help to
> determine if the Knox is working harder during the SSL transfer.
>
> Kevin Risden
>
>
> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
>
>> I'm just using curl to download a single large file. So I suspect SSL
>> handshake just happens once?
>>
>> On Tue, Sep 11, 2018 at 12:02 PM
>> Kevin Risden
>> <kr...@apache.org> wrote:
>>
>>> What client are you using to connect Knox? Is this for a single file or
>>> a bunch of files?
>>>
>>> The SSL handshake can be slow if the client doesn't keep the connection
>>> open.
>>>
>>> Kevin Risden
>>>
>>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>>
>>>> Thanks Larry. But the only difference is this part in my
>>>> gateway-site.xml.
>>>>
>>>> *<property>*
>>>> * <name>ssl.enabled</name>*
>>>> * <value>false</value>*
>>>> * <description>Indicates whether SSL is enabled.</description>*
>>>> *</property>*
>>>>
>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org>
>>>> wrote:
>>>>
>>>>> I really don't think that kind of difference should be expected from
>>>>> merely SSL overhead.
>>>>> I don't however have any metrics to contradict it either since I do
>>>>> not run Knox without SSL.
>>>>>
>>>>> Given the above, I am struggling coming up with a meaningful response
>>>>> to this. :(
>>>>> I don't think you should see a 10 fold increase in speed by disabling
>>>>> SSL though.
>>>>>
>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>>>
>>>>>> Any idea guys?
>>>>>>
>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>>>>
>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which
>>>>>>> is because of encrypted over SSL.
>>>>>>>
>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>>>>>> does surprised me with such a difference. Is it normal from your
>>>>>>> perspective?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Guang
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>>>
>>>>>>>> Hi Guang,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Keep in mind the data is being encrypted over SSL. If you disable
>>>>>>>> SSL you will most likely see a very significant boost in throughput. Some
>>>>>>>> people have used more powerful computers to make encryption quicker.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> David
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Guang – This is somewhat to be expected.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>>>> the source.
>>>>>>>>
>>>>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>>>> capacity since only 1 host is serving data to you.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Also, if running on a cloud provider, the Knox host may be a
>>>>>>>> smaller instance size with lower network capacity.
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Sean Roberts
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a
>>>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>>>> least.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Guang
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
Hmmm yea curl for a single file should do the handshake once.
What are the system performance statistics during the SSL vs non SSL
testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
if using that. Otherwise watching top may be helpful. It would be help to
determine if the Knox is working harder during the SSL transfer.
Kevin Risden
On Wed, Sep 12, 2018 at 2:52 PM Guang Yang <ko...@uber.com> wrote:
> I'm just using curl to download a single large file. So I suspect SSL
> handshake just happens once?
>
> On Tue, Sep 11, 2018 at 12:02 PM
> Kevin Risden
> <kr...@apache.org> wrote:
>
>> What client are you using to connect Knox? Is this for a single file or a
>> bunch of files?
>>
>> The SSL handshake can be slow if the client doesn't keep the connection
>> open.
>>
>> Kevin Risden
>>
>> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>>
>>> Thanks Larry. But the only difference is this part in my
>>> gateway-site.xml.
>>>
>>> *<property>*
>>> * <name>ssl.enabled</name>*
>>> * <value>false</value>*
>>> * <description>Indicates whether SSL is enabled.</description>*
>>> *</property>*
>>>
>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>>>
>>>> I really don't think that kind of difference should be expected from
>>>> merely SSL overhead.
>>>> I don't however have any metrics to contradict it either since I do not
>>>> run Knox without SSL.
>>>>
>>>> Given the above, I am struggling coming up with a meaningful response
>>>> to this. :(
>>>> I don't think you should see a 10 fold increase in speed by disabling
>>>> SSL though.
>>>>
>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>>
>>>>> Any idea guys?
>>>>>
>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>>>
>>>>>> Thanks guys! The issue seems exactly what David pointed out, which is
>>>>>> because of encrypted over SSL.
>>>>>>
>>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>>>>> does surprised me with such a difference. Is it normal from your
>>>>>> perspective?
>>>>>>
>>>>>> Thanks,
>>>>>> Guang
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>>
>>>>>>> Hi Guang,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Keep in mind the data is being encrypted over SSL. If you disable
>>>>>>> SSL you will most likely see a very significant boost in throughput. Some
>>>>>>> people have used more powerful computers to make encryption quicker.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> David
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Guang – This is somewhat to be expected.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>>> the source.
>>>>>>>
>>>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>>> capacity since only 1 host is serving data to you.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>>>>>> instance size with lower network capacity.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Sean Roberts
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a
>>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>>> least.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Guang
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
Re: WebHDFS performance issue in Knox
Posted by Guang Yang <ko...@uber.com>.
I'm just using curl to download a single large file. So I suspect SSL
handshake just happens once?
On Tue, Sep 11, 2018 at 12:02 PM Kevin Risden <kr...@apache.org> wrote:
> What client are you using to connect Knox? Is this for a single file or a
> bunch of files?
>
> The SSL handshake can be slow if the client doesn't keep the connection
> open.
>
> Kevin Risden
>
> On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
>
>> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>>
>> *<property>*
>> * <name>ssl.enabled</name>*
>> * <value>false</value>*
>> * <description>Indicates whether SSL is enabled.</description>*
>> *</property>*
>>
>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>>
>>> I really don't think that kind of difference should be expected from
>>> merely SSL overhead.
>>> I don't however have any metrics to contradict it either since I do not
>>> run Knox without SSL.
>>>
>>> Given the above, I am struggling coming up with a meaningful response to
>>> this. :(
>>> I don't think you should see a 10 fold increase in speed by disabling
>>> SSL though.
>>>
>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>>
>>>> Any idea guys?
>>>>
>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>>
>>>>> Thanks guys! The issue seems exactly what David pointed out, which is
>>>>> because of encrypted over SSL.
>>>>>
>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>>>> does surprised me with such a difference. Is it normal from your
>>>>> perspective?
>>>>>
>>>>> Thanks,
>>>>> Guang
>>>>>
>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>> dvillarreal@hortonworks.com> wrote:
>>>>>
>>>>>> Hi Guang,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Keep in mind the data is being encrypted over SSL. If you disable
>>>>>> SSL you will most likely see a very significant boost in throughput. Some
>>>>>> people have used more powerful computers to make encryption quicker.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>>
>>>>>> David
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>>
>>>>>>
>>>>>>
>>>>>> Guang – This is somewhat to be expected.
>>>>>>
>>>>>>
>>>>>>
>>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>>> request across many data nodes. Also, you are getting data directly from
>>>>>> the source.
>>>>>>
>>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>>> capacity since only 1 host is serving data to you.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>>>>> instance size with lower network capacity.
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Sean Roberts
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a
>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s.
>>>>>> However, if we download directly from datanode, the speed is about 40M/s at
>>>>>> least.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Guang
>>>>>>
>>>>>
>>>>>
>>>>
>>
Re: WebHDFS performance issue in Knox
Posted by Kevin Risden <kr...@apache.org>.
What client are you using to connect Knox? Is this for a single file or a
bunch of files?
The SSL handshake can be slow if the client doesn't keep the connection
open.
Kevin Risden
On Tue, Sep 11, 2018, 14:51 Guang Yang <ko...@uber.com> wrote:
> Thanks Larry. But the only difference is this part in my gateway-site.xml.
>
> *<property>*
> * <name>ssl.enabled</name>*
> * <value>false</value>*
> * <description>Indicates whether SSL is enabled.</description>*
> *</property>*
>
> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
>
>> I really don't think that kind of difference should be expected from
>> merely SSL overhead.
>> I don't however have any metrics to contradict it either since I do not
>> run Knox without SSL.
>>
>> Given the above, I am struggling coming up with a meaningful response to
>> this. :(
>> I don't think you should see a 10 fold increase in speed by disabling SSL
>> though.
>>
>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>>
>>> Any idea guys?
>>>
>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>>
>>>> Thanks guys! The issue seems exactly what David pointed out, which is
>>>> because of encrypted over SSL.
>>>>
>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>>> does surprised me with such a difference. Is it normal from your
>>>> perspective?
>>>>
>>>> Thanks,
>>>> Guang
>>>>
>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>> dvillarreal@hortonworks.com> wrote:
>>>>
>>>>> Hi Guang,
>>>>>
>>>>>
>>>>>
>>>>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>>>>> you will most likely see a very significant boost in throughput. Some
>>>>> people have used more powerful computers to make encryption quicker.
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>>
>>>>>
>>>>>
>>>>> Guang – This is somewhat to be expected.
>>>>>
>>>>>
>>>>>
>>>>> When you talk to WebHDFS directly, the client can distribute the
>>>>> request across many data nodes. Also, you are getting data directly from
>>>>> the source.
>>>>>
>>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>>> responsible for fetching from the datanodes and consolidating to send to
>>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>>> capacity since only 1 host is serving data to you.
>>>>>
>>>>>
>>>>>
>>>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>>>> instance size with lower network capacity.
>>>>>
>>>>> --
>>>>>
>>>>> Sean Roberts
>>>>>
>>>>>
>>>>>
>>>>> *From: *Guang Yang <ko...@uber.com>
>>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>>> *Subject: *WebHDFS performance issue in Knox
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>>>>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>>>>> if we download directly from datanode, the speed is about 40M/s at least.
>>>>>
>>>>>
>>>>>
>>>>> Are you guys aware of this problem? Any suggestion?
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Guang
>>>>>
>>>>
>>>>
>>>
>
Re: WebHDFS performance issue in Knox
Posted by Guang Yang <ko...@uber.com>.
Thanks Larry. But the only difference is this part in my gateway-site.xml.
*<property>*
* <name>ssl.enabled</name>*
* <value>false</value>*
* <description>Indicates whether SSL is enabled.</description>*
*</property>*
On Tue, Sep 11, 2018 at 11:42 AM, larry mccay <lm...@apache.org> wrote:
> I really don't think that kind of difference should be expected from
> merely SSL overhead.
> I don't however have any metrics to contradict it either since I do not
> run Knox without SSL.
>
> Given the above, I am struggling coming up with a meaningful response to
> this. :(
> I don't think you should see a 10 fold increase in speed by disabling SSL
> though.
>
> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
>
>> Any idea guys?
>>
>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>>
>>> Thanks guys! The issue seems exactly what David pointed out, which is
>>> because of encrypted over SSL.
>>>
>>> Without Knox, the download speed can reach to *400M/s* if I call
>>> Namenode directly. And with disabling SSL, the speed can reach to
>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>> significantly to *~40M/s*. I know it's because of encrypted, but it
>>> does surprised me with such a difference. Is it normal from your
>>> perspective?
>>>
>>> Thanks,
>>> Guang
>>>
>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>> dvillarreal@hortonworks.com> wrote:
>>>
>>>> Hi Guang,
>>>>
>>>>
>>>>
>>>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>>>> you will most likely see a very significant boost in throughput. Some
>>>> people have used more powerful computers to make encryption quicker.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>>
>>>>
>>>>
>>>> Guang – This is somewhat to be expected.
>>>>
>>>>
>>>>
>>>> When you talk to WebHDFS directly, the client can distribute the
>>>> request across many data nodes. Also, you are getting data directly from
>>>> the source.
>>>>
>>>> With Knox, all traffic goes through the single Knox host. Knox is
>>>> responsible for fetching from the datanodes and consolidating to send to
>>>> you. This means overhead as it’s acting as a middle man, and lower network
>>>> capacity since only 1 host is serving data to you.
>>>>
>>>>
>>>>
>>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>>> instance size with lower network capacity.
>>>>
>>>> --
>>>>
>>>> Sean Roberts
>>>>
>>>>
>>>>
>>>> *From: *Guang Yang <ko...@uber.com>
>>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>>> *Subject: *WebHDFS performance issue in Knox
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>>>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>>>> if we download directly from datanode, the speed is about 40M/s at least.
>>>>
>>>>
>>>>
>>>> Are you guys aware of this problem? Any suggestion?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Guang
>>>>
>>>
>>>
>>
Re: WebHDFS performance issue in Knox
Posted by larry mccay <lm...@apache.org>.
I really don't think that kind of difference should be expected from merely
SSL overhead.
I don't however have any metrics to contradict it either since I do not run
Knox without SSL.
Given the above, I am struggling coming up with a meaningful response to
this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL
though.
On Tue, Sep 11, 2018 at 2:35 PM Guang Yang <ko...@uber.com> wrote:
> Any idea guys?
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
>
>> Thanks guys! The issue seems exactly what David pointed out, which is
>> because of encrypted over SSL.
>>
>> Without Knox, the download speed can reach to *400M/s* if I call
>> Namenode directly. And with disabling SSL, the speed can reach to
>> *~400M/s* as well through Knox. But with SSL, the speed drops
>> significantly to *~40M/s*. I know it's because of encrypted, but it does
>> surprised me with such a difference. Is it normal from your perspective?
>>
>> Thanks,
>> Guang
>>
>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>> dvillarreal@hortonworks.com> wrote:
>>
>>> Hi Guang,
>>>
>>>
>>>
>>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>>> you will most likely see a very significant boost in throughput. Some
>>> people have used more powerful computers to make encryption quicker.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>> *From: *Sean Roberts <sr...@hortonworks.com>
>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>
>>>
>>>
>>> Guang – This is somewhat to be expected.
>>>
>>>
>>>
>>> When you talk to WebHDFS directly, the client can distribute the request
>>> across many data nodes. Also, you are getting data directly from the source.
>>>
>>> With Knox, all traffic goes through the single Knox host. Knox is
>>> responsible for fetching from the datanodes and consolidating to send to
>>> you. This means overhead as it’s acting as a middle man, and lower network
>>> capacity since only 1 host is serving data to you.
>>>
>>>
>>>
>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>> instance size with lower network capacity.
>>>
>>> --
>>>
>>> Sean Roberts
>>>
>>>
>>>
>>> *From: *Guang Yang <ko...@uber.com>
>>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>>> *Subject: *WebHDFS performance issue in Knox
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>>> if we download directly from datanode, the speed is about 40M/s at least.
>>>
>>>
>>>
>>> Are you guys aware of this problem? Any suggestion?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Guang
>>>
>>
>>
>
Re: WebHDFS performance issue in Knox
Posted by Guang Yang <ko...@uber.com>.
Any idea guys?
On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang <ko...@uber.com> wrote:
> Thanks guys! The issue seems exactly what David pointed out, which is
> because of encrypted over SSL.
>
> Without Knox, the download speed can reach to *400M/s* if I call Namenode
> directly. And with disabling SSL, the speed can reach to *~400M/s* as
> well through Knox. But with SSL, the speed drops significantly to *~40M/s*.
> I know it's because of encrypted, but it does surprised me with such a
> difference. Is it normal from your perspective?
>
> Thanks,
> Guang
>
> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
> dvillarreal@hortonworks.com> wrote:
>
>> Hi Guang,
>>
>>
>>
>> Keep in mind the data is being encrypted over SSL. If you disable SSL
>> you will most likely see a very significant boost in throughput. Some
>> people have used more powerful computers to make encryption quicker.
>>
>>
>>
>> Thanks,
>>
>>
>>
>> David
>>
>>
>>
>> *From: *Sean Roberts <sr...@hortonworks.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> Guang – This is somewhat to be expected.
>>
>>
>>
>> When you talk to WebHDFS directly, the client can distribute the request
>> across many data nodes. Also, you are getting data directly from the source.
>>
>> With Knox, all traffic goes through the single Knox host. Knox is
>> responsible for fetching from the datanodes and consolidating to send to
>> you. This means overhead as it’s acting as a middle man, and lower network
>> capacity since only 1 host is serving data to you.
>>
>>
>>
>> Also, if running on a cloud provider, the Knox host may be a smaller
>> instance size with lower network capacity.
>>
>> --
>>
>> Sean Roberts
>>
>>
>>
>> *From: *Guang Yang <ko...@uber.com>
>> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Date: *Tuesday, 4 September 2018 at 07:46
>> *To: *"user@knox.apache.org" <us...@knox.apache.org>
>> *Subject: *WebHDFS performance issue in Knox
>>
>>
>>
>> Hi,
>>
>>
>>
>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>> if we download directly from datanode, the speed is about 40M/s at least.
>>
>>
>>
>> Are you guys aware of this problem? Any suggestion?
>>
>>
>>
>> Thanks,
>>
>> Guang
>>
>
>
Re: WebHDFS performance issue in Knox
Posted by Guang Yang <ko...@uber.com>.
Thanks guys! The issue seems exactly what David pointed out, which is
because of encrypted over SSL.
Without Knox, the download speed can reach to *400M/s* if I call Namenode
directly. And with disabling SSL, the speed can reach to *~400M/s* as well
through Knox. But with SSL, the speed drops significantly to *~40M/s*. I
know it's because of encrypted, but it does surprised me with such a
difference. Is it normal from your perspective?
Thanks,
Guang
On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
dvillarreal@hortonworks.com> wrote:
> Hi Guang,
>
>
>
> Keep in mind the data is being encrypted over SSL. If you disable SSL you
> will most likely see a very significant boost in throughput. Some people
> have used more powerful computers to make encryption quicker.
>
>
>
> Thanks,
>
>
>
> David
>
>
>
> *From: *Sean Roberts <sr...@hortonworks.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, September 4, 2018 at 1:53 AM
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Guang – This is somewhat to be expected.
>
>
>
> When you talk to WebHDFS directly, the client can distribute the request
> across many data nodes. Also, you are getting data directly from the source.
>
> With Knox, all traffic goes through the single Knox host. Knox is
> responsible for fetching from the datanodes and consolidating to send to
> you. This means overhead as it’s acting as a middle man, and lower network
> capacity since only 1 host is serving data to you.
>
>
>
> Also, if running on a cloud provider, the Knox host may be a smaller
> instance size with lower network capacity.
>
> --
>
> Sean Roberts
>
>
>
> *From: *Guang Yang <ko...@uber.com>
> *Reply-To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Date: *Tuesday, 4 September 2018 at 07:46
> *To: *"user@knox.apache.org" <us...@knox.apache.org>
> *Subject: *WebHDFS performance issue in Knox
>
>
>
> Hi,
>
>
>
> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
> through WebHDFS in Knox, the download speed is just about 11M/s. However,
> if we download directly from datanode, the speed is about 40M/s at least.
>
>
>
> Are you guys aware of this problem? Any suggestion?
>
>
>
> Thanks,
>
> Guang
>
Re: WebHDFS performance issue in Knox
Posted by David Villarreal <dv...@hortonworks.com>.
Hi Guang,
Keep in mind the data is being encrypted over SSL. If you disable SSL you will most likely see a very significant boost in throughput. Some people have used more powerful computers to make encryption quicker.
Thanks,
David
From: Sean Roberts <sr...@hortonworks.com>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, September 4, 2018 at 1:53 AM
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: Re: WebHDFS performance issue in Knox
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang
Re: WebHDFS performance issue in Knox
Posted by Sean Roberts <sr...@hortonworks.com>.
Guang – This is somewhat to be expected.
When you talk to WebHDFS directly, the client can distribute the request across many data nodes. Also, you are getting data directly from the source.
With Knox, all traffic goes through the single Knox host. Knox is responsible for fetching from the datanodes and consolidating to send to you. This means overhead as it’s acting as a middle man, and lower network capacity since only 1 host is serving data to you.
Also, if running on a cloud provider, the Knox host may be a smaller instance size with lower network capacity.
--
Sean Roberts
From: Guang Yang <ko...@uber.com>
Reply-To: "user@knox.apache.org" <us...@knox.apache.org>
Date: Tuesday, 4 September 2018 at 07:46
To: "user@knox.apache.org" <us...@knox.apache.org>
Subject: WebHDFS performance issue in Knox
Hi,
We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file through WebHDFS in Knox, the download speed is just about 11M/s. However, if we download directly from datanode, the speed is about 40M/s at least.
Are you guys aware of this problem? Any suggestion?
Thanks,
Guang