You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by =?iso-2022-cn?B?GyQpQQ5SV1VxOyoP?= <yi...@longrise.com.cn> on 2018/03/24 07:57:53 UTC
geode server network request is very slow!
Hello User,
my question is network request is very slow ,only 5000/second ,
how can i fix it???
ths problem is my server is CentOS7 ,hardware : CPU 8700K ,
memory:32G , disk:intel 760p 256G , network : 1000M
and i start geode 1.4.0 , use gfsh command :
start locator --name=locator1 --port=30000 --bind-address=192.168.0.226 --http-service-port=0 --http-service-bind-address=192.168.0.226 --properties-file=./config/server.properties --J=-Dp2p.handshakeTimeoutMs=75000
configure pdx --disk-store
start server --name=server1 --server-port=0 --locators=192.168.0.226[30000] --bind-address=192.168.0.226 --J=-Dgemfire.ALLOW_PERSISTENT_TRANSACTIONS=true --force=1 --properties-file=./config/server.properties --groups=leap,lwfp,portal --J=-Dp2p.handshakeTimeoutMs=75000
create region --name=r1 --type=REPLICATE
1)at the server pc , run java code , the region entry size is 0 ,
get region entry , speed is 63000/second
Properties props = new Properties();
ClientCacheFactory cf = new ClientCacheFactory(props);
ClientCache cache = cf.addPoolLocator("192.168.0.226", 30000).set("log-level", "WARN").create();
Region<String, String> region =
cache.<String, String> createClientRegionFactory(ClientRegionShortcut.CACHING_PROXY).create("r1");
final AtomicInteger counter = new AtomicInteger(0);
final AtomicInteger idx = new AtomicInteger(0);
for (int i = 0; i < 1; i++)
{
new Thread(new Runnable()
{
public void run ()
{
while (true)
{
region.get(idx.getAndIncrement());
counter.incrementAndGet();
}
}
}).start();
}
2)but at the other pc , at same network , ip address
192.168.0.169 , same 1000M network , the same code , get region entry from the
192.168.0.226 server , speed is 5000/second
note : sysctl.conf
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096
kernel.sem= 500 64000 50 400
net.core.netdev_max_backlog=30000
net.core.wmem_max=67108864
net.core.rmem_max=67108864
net.ipv4.tcp_congestion_control=htcp
net.ipv4.tcp_congestion_window=10
net.ipv4.tcp_fin_timeout=10
net.ipv4.tcp_keepalive_interval=30
net.ipv4.tcp_keepalive_probes=5
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_max_orphans=16384
net.ipv4.tcp_max_tw_buckets=1440000
net.ipv4.tcp_no_metrics_save=1
net.ipv4.tcp_orphan_retries=0
net.ipv4.tcp_rfc1337=1
net.ipv4.tcp_rmem=10240 131072 33554432
net.ipv4.tcp_wmem=10240 131072 33554432
net.ipv4.tcp_sack=1
net.ipv4.tcp_slow_start_after_idle=0
net.ipv4.tcp_syncookies=0
net.ipv4.tcp_max_syn_backlog="16384"
net.ipv4.tcp_synack_retries="1"
net.ipv4.tcp_max_orphans="400000"
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_window_scaling=1
--
Best regards,
$)ARWUq;* mailto:yizh@longrise.com.cn
--
Best regards,
$)ARWUq;* mailto:yizh@longrise.com.cn
Re: geode server network request is very slow!
Posted by Charlie Black <cb...@pivotal.io>.
Don’t forget about the physical nature of a distributed call. The local
client get doesn’t go through the switch and the remote client goes through
the switch.
As a quick check try a simple ping from one machine to the other and ping
local host. Notice how much quicker the local ping to the remote host is.
Then check how those numbers correlate to the performance you are getting
out of geode.
If you would like to aggregate the cost of the network operations - geode
has a putAll / getAll / removeAll.
Regards,
Charlie
On Sat, Mar 24, 2018 at 8:40 AM Jens Deppe <jd...@pivotal.io> wrote:
> Hi,
>
> One other quick check would be to see if you have any network errors
> showing up in the output of 'ifconfig' on the client PC.
>
> For tuning take a look at
> https://cwiki.apache.org/confluence/display/GEODE/Network+Configuration+Best+Practices
>
> There are a lot of possibilities there, but I would suggest focusing
> on net.ipv4.tcp_rmem and net.ipv4.tcp_wmem and increase those to 32MB as
> suggested.
>
> --Jens
>
>
> On Sat, Mar 24, 2018 at 12:57 AM, 易振华 <yi...@longrise.com.cn> wrote:
>
>> Hello User,
>>
>> my question is network request is very slow ,only 5000/second ,
>> how can i fix it???
>>
>> ths problem is my server is CentOS7 ,hardware : CPU 8700K ,
>> memory:32G , disk:intel 760p 256G , network : 1000M
>>
>> and i start geode 1.4.0 , use gfsh command :
>> start locator --name=locator1 --port=30000
>> --bind-address=192.168.0.226 --http-service-port=0
>> --http-service-bind-address=192.168.0.226
>> --properties-file=./config/server.properties
>> --J=-Dp2p.handshakeTimeoutMs=75000
>>
>> configure pdx --disk-store
>>
>> start server --name=server1 --server-port=0
>> --locators=192.168.0.226[30000] --bind-address=192.168.0.226
>> --J=-Dgemfire.ALLOW_PERSISTENT_TRANSACTIONS=true --force=1
>> --properties-file=./config/server.properties --groups=leap,lwfp,portal
>> --J=-Dp2p.handshakeTimeoutMs=75000
>>
>> create region --name=r1 --type=REPLICATE
>>
>> 1)at the server pc , run java code , the region entry size is 0 ,
>> get region entry , speed is 63000/second
>>
>> Properties props = new Properties();
>> ClientCacheFactory cf = new ClientCacheFactory(props);
>> ClientCache cache = cf.addPoolLocator("192.168.0.226",
>> 30000).set("log-level", "WARN").create();
>> Region<String, String> region =
>> cache.<String, String>
>> createClientRegionFactory(ClientRegionShortcut.CACHING_PROXY).create("r1");
>> final AtomicInteger counter = new AtomicInteger(0);
>> final AtomicInteger idx = new AtomicInteger(0);
>> for (int i = 0; i < 1; i++)
>> {
>> new Thread(new Runnable()
>> {
>> public void run ()
>> {
>> while (true)
>> {
>> region.get(idx.getAndIncrement());
>> counter.incrementAndGet();
>> }
>> }
>> }).start();
>> }
>>
>> 2)but at the other pc , at same network , ip address
>> 192.168.0.169 , same 1000M network , the same code , get region entry
>> from the
>> 192.168.0.226 server , speed is 5000/second
>>
>>
>>
>> note : sysctl.conf
>>
>> kernel.sysrq = 0
>> kernel.core_uses_pid = 1
>> kernel.msgmnb = 65536
>> kernel.msgmax = 65536
>> kernel.shmmax = 68719476736
>> kernel.shmall = 4294967296
>>
>>
>>
>>
>> fs.aio-max-nr = 1048576
>> fs.file-max = 6815744
>> kernel.shmall = 2097152
>> kernel.shmmax = 4294967295
>> kernel.shmmni = 4096
>> kernel.sem= 500 64000 50 400
>>
>>
>> net.core.netdev_max_backlog=30000
>> net.core.wmem_max=67108864
>> net.core.rmem_max=67108864
>> net.ipv4.tcp_congestion_control=htcp
>> net.ipv4.tcp_congestion_window=10
>> net.ipv4.tcp_fin_timeout=10
>> net.ipv4.tcp_keepalive_interval=30
>> net.ipv4.tcp_keepalive_probes=5
>> net.ipv4.tcp_keepalive_time=600
>> net.ipv4.tcp_low_latency=1
>> net.ipv4.tcp_max_orphans=16384
>> net.ipv4.tcp_max_tw_buckets=1440000
>> net.ipv4.tcp_no_metrics_save=1
>> net.ipv4.tcp_orphan_retries=0
>> net.ipv4.tcp_rfc1337=1
>> net.ipv4.tcp_rmem=10240 131072 33554432
>> net.ipv4.tcp_wmem=10240 131072 33554432
>> net.ipv4.tcp_sack=1
>> net.ipv4.tcp_slow_start_after_idle=0
>> net.ipv4.tcp_syncookies=0
>> net.ipv4.tcp_max_syn_backlog="16384"
>> net.ipv4.tcp_synack_retries="1"
>> net.ipv4.tcp_max_orphans="400000"
>> net.ipv4.tcp_timestamps=1
>> net.ipv4.tcp_tw_recycle=1
>> net.ipv4.tcp_tw_reuse=1
>> net.ipv4.tcp_window_scaling=1
>>
>>
>>
>>
>>
>> --
>> Best regards,
>> 易振华 mailto:yizh@longrise.com.cn
>>
>> --
>> Best regards,
>> 易振华 mailto:yizh@longrise.com.cn
>>
>>
> --
cblack@pivotal.io | +1.858.480.9722
Re: geode server network request is very slow!
Posted by Jens Deppe <jd...@pivotal.io>.
Hi,
One other quick check would be to see if you have any network errors
showing up in the output of 'ifconfig' on the client PC.
For tuning take a look at
https://cwiki.apache.org/confluence/display/GEODE/Network+Configuration+Best+Practices
There are a lot of possibilities there, but I would suggest focusing
on net.ipv4.tcp_rmem and net.ipv4.tcp_wmem and increase those to 32MB as
suggested.
--Jens
On Sat, Mar 24, 2018 at 12:57 AM, 易振华 <yi...@longrise.com.cn> wrote:
> Hello User,
>
> my question is network request is very slow ,only 5000/second ,
> how can i fix it???
>
> ths problem is my server is CentOS7 ,hardware : CPU 8700K ,
> memory:32G , disk:intel 760p 256G , network : 1000M
>
> and i start geode 1.4.0 , use gfsh command :
> start locator --name=locator1 --port=30000
> --bind-address=192.168.0.226 --http-service-port=0
> --http-service-bind-address=192.168.0.226 --properties-file=./config/server.properties
> --J=-Dp2p.handshakeTimeoutMs=75000
>
> configure pdx --disk-store
>
> start server --name=server1 --server-port=0
> --locators=192.168.0.226[30000] --bind-address=192.168.0.226
> --J=-Dgemfire.ALLOW_PERSISTENT_TRANSACTIONS=true --force=1
> --properties-file=./config/server.properties --groups=leap,lwfp,portal
> --J=-Dp2p.handshakeTimeoutMs=75000
>
> create region --name=r1 --type=REPLICATE
>
> 1)at the server pc , run java code , the region entry size is 0 ,
> get region entry , speed is 63000/second
>
> Properties props = new Properties();
> ClientCacheFactory cf = new ClientCacheFactory(props);
> ClientCache cache = cf.addPoolLocator("192.168.0.226",
> 30000).set("log-level", "WARN").create();
> Region<String, String> region =
> cache.<String, String> createClientRegionFactory(
> ClientRegionShortcut.CACHING_PROXY).create("r1");
> final AtomicInteger counter = new AtomicInteger(0);
> final AtomicInteger idx = new AtomicInteger(0);
> for (int i = 0; i < 1; i++)
> {
> new Thread(new Runnable()
> {
> public void run ()
> {
> while (true)
> {
> region.get(idx.getAndIncrement());
> counter.incrementAndGet();
> }
> }
> }).start();
> }
>
> 2)but at the other pc , at same network , ip address
> 192.168.0.169 , same 1000M network , the same code , get region entry
> from the
> 192.168.0.226 server , speed is 5000/second
>
>
>
> note : sysctl.conf
>
> kernel.sysrq = 0
> kernel.core_uses_pid = 1
> kernel.msgmnb = 65536
> kernel.msgmax = 65536
> kernel.shmmax = 68719476736
> kernel.shmall = 4294967296
>
>
>
>
> fs.aio-max-nr = 1048576
> fs.file-max = 6815744
> kernel.shmall = 2097152
> kernel.shmmax = 4294967295
> kernel.shmmni = 4096
> kernel.sem= 500 64000 50 400
>
>
> net.core.netdev_max_backlog=30000
> net.core.wmem_max=67108864
> net.core.rmem_max=67108864
> net.ipv4.tcp_congestion_control=htcp
> net.ipv4.tcp_congestion_window=10
> net.ipv4.tcp_fin_timeout=10
> net.ipv4.tcp_keepalive_interval=30
> net.ipv4.tcp_keepalive_probes=5
> net.ipv4.tcp_keepalive_time=600
> net.ipv4.tcp_low_latency=1
> net.ipv4.tcp_max_orphans=16384
> net.ipv4.tcp_max_tw_buckets=1440000
> net.ipv4.tcp_no_metrics_save=1
> net.ipv4.tcp_orphan_retries=0
> net.ipv4.tcp_rfc1337=1
> net.ipv4.tcp_rmem=10240 131072 33554432
> net.ipv4.tcp_wmem=10240 131072 33554432
> net.ipv4.tcp_sack=1
> net.ipv4.tcp_slow_start_after_idle=0
> net.ipv4.tcp_syncookies=0
> net.ipv4.tcp_max_syn_backlog="16384"
> net.ipv4.tcp_synack_retries="1"
> net.ipv4.tcp_max_orphans="400000"
> net.ipv4.tcp_timestamps=1
> net.ipv4.tcp_tw_recycle=1
> net.ipv4.tcp_tw_reuse=1
> net.ipv4.tcp_window_scaling=1
>
>
>
>
>
> --
> Best regards,
> 易振华 mailto:yizh@longrise.com.cn
>
> --
> Best regards,
> 易振华 mailto:yizh@longrise.com.cn
>
>