You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by "Xu, Nan" <nx...@baml.com> on 2017/06/08 21:09:53 UTC
Geode Performance how to push the cpu to 100%?
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Galen M O'Sullivan <go...@pivotal.io>.
Hi Nan,
93% seems like pretty good CPU utilization. I can't speculate as to exactly
what the rest of it is being used for, and I don't know exactly how
performance metrics are gathered, but you won't generally get 100% for
reasons I don't understand that well either. I would speculate that factors
could be OS usage, timeslices, I/O, interrupts, the way stats are gathered
and the time that is used gathering stats themselves, locking... but I
don't know. I think throughput numbers are probably more meaningful than
CPU percentage usage.
Best,
Galen
On Thu, Jun 8, 2017 at 5:13 PM, Akihiro Kitada <ak...@pivotal.io> wrote:
> Hello Nan,
>
> Why don't you check Geode statistics
>
> http://geode.apache.org/docs/guide/11/reference/statistics/
> statistics_list.html
>
> Maybe, disk I/O or some other causes could be involved...
>
>
>
> --
> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
> <+81%2080-3716-3736>
> Support.Pivotal.io <http://support.pivotal.io/> | Mon-Fri 9:00am to
> 5:30pm JST | 1-877-477-2269 <(877)%20477-2269>
> [image: support] <https://support.pivotal.io/> [image: twitter]
> <https://twitter.com/pivotal> [image: linkedin]
> <https://www.linkedin.com/company/3048967> [image: facebook]
> <https://www.facebook.com/pivotalsoftware> [image: google plus]
> <https://plus.google.com/+Pivotal> [image: youtube]
> <https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl>
>
>
> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
>
>> Hi,
>>
>>
>>
>> I am trying to see the limit of geode performance.
>>
>>
>>
>> Here is what I did.
>>
>>
>>
>> Single machine: OS: centos 7. 8 core. 2.6G.
>>
>>
>>
>> Create a single locator and a single server and a single region.
>>
>>
>>
>> Only configuration is
>>
>> Server conserve-sockets= false
>>
>> Region is PARTITION
>>
>> Client and server running on the same machine.
>>
>>
>>
>>
>>
>> In my client. I setup a 16 thread pool to get data. But I can only push
>> the cpu to around 90—93% on the centos.
>>
>>
>>
>> Why I cannot push it to 100%? I am suspecting
>>
>> 1. The tcp connection between the server and client is not fast
>> enough. Maybe increase the number of tcp connection? I only see one
>> connection between client and server.
>>
>> 2. There are some lock at the server? I realize that I can push
>> the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
>> so maybe there are some other setting I am missing.
>>
>>
>>
>> Thanks,
>>
>> Nan
>>
>>
>>
>> Client side program.
>>
>>
>>
>> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
>> 10334)
>>
>> .set("log-level", "info")
>>
>> .set("conserve-sockets", "false")
>>
>> .setPoolMinConnections(4)
>>
>> .setPoolMaxConnections(12)
>>
>> .setPoolThreadLocalConnections(true)
>>
>> .create
>>
>>
>>
>> val regionFactory: ClientRegionFactory[String, String] =
>> cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
>>
>> val region1: Region[String, String] = regionFactory.create(region)
>>
>>
>>
>> implicit val ec = ExecutionContext.fromExecutorS
>> ervice(Executors.newFixedThreadPool(16))
>>
>> var j = new AtomicLong(0)
>>
>> for (i <- 1 to 16) yield Future {
>>
>> while(true){
>>
>> val cj = j.addAndGet(1)
>>
>> region1.get("" + (rnd.nextInt(2000) + 1))
>>
>> }
>>
>> }
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.bankofamerica.com/emaildisclaimer. If you are not the
>> intended recipient, please delete this message.
>>
>
>
Re: Geode Performance how to push the cpu to 100%?
Posted by Jacob Barrett <jb...@pivotal.io>.
What OS? You may need to tune loopback interface to handle the throughput. If you're putting 50k messages / sec you can consider that roughly 100k packets per sec. Is the system tuned for that? If it isn't tuned it could be that you're hitting a bottleneck there and unable to saturate the system.
On a more broad note though this isn't the best way to gauge performance. You really want to gauge what your configuration can do a production levels, that is some level less than fully saturated. I would target somewhere between 50% -75% depending on how much burst capacity you would like. You would never want to go to production where the system is running above 80% constantly because you have very little room for bursting and will easily tip over the system with the slightest hiccups.
-Jake
Sent from my iPhone
> On Jun 9, 2017, at 9:15 AM, Xu, Nan <nx...@baml.com> wrote:
>
> Thanks for pointing this out, very useful.
>
> I change the way I test the performance and get a result I can NOT explain.
>
> I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
>
> The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
>
> I can see there are 128 tcp connections between them and I send about 50,000 message/s
>
> Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
>
> Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
>
> Thanks,
> Nan
>
> From: Akihiro Kitada [mailto:akitada@pivotal.io]
> Sent: Thursday, June 08, 2017 7:13 PM
> To: user@geode.apache.org
> Subject: Re: Geode Performance how to push the cpu to 100%?
>
> Hello Nan,
>
> Why don't you check Geode statistics
>
> http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html
>
> Maybe, disk I/O or some other causes could be involved...
>
>
>
> --
> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
> Support.Pivotal.io | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269
>
>
>
> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
> Hi,
>
> I am trying to see the limit of geode performance.
>
> Here is what I did.
>
> Single machine: OS: centos 7. 8 core. 2.6G.
>
> Create a single locator and a single server and a single region.
>
> Only configuration is
> Server conserve-sockets= false
> Region is PARTITION
> Client and server running on the same machine.
>
>
> In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
>
> Why I cannot push it to 100%? I am suspecting
> 1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
>
> 2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
>
>
> Thanks,
> Nan
>
> Client side program.
>
> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
> .set("log-level", "info")
> .set("conserve-sockets", "false")
> .setPoolMinConnections(4)
> .setPoolMaxConnections(12)
> .setPoolThreadLocalConnections(true)
> .create
>
> val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
> val region1: Region[String, String] = regionFactory.create(region)
>
> implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
> var j = new AtomicLong(0)
> for (i <- 1 to 16) yield Future {
> while(true){
> val cj = j.addAndGet(1)
> region1.get("" + (rnd.nextInt(2000) + 1))
> }
> }
>
>
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Absolutely right, did a iperf and thought I still had unused bandwidth. While, iperf is bit/sec and resource manager is byte/sec. bandwidth was actually used up. I am sending around 180M bytes/sec.
Sent with BlackBerry Work (www.blackberry.com)
________________________________
From: Anthony Baker <ab...@pivotal.io>
Sent: Jun 10, 2017 12:19 PM
To: user@geode.apache.org
Subject: Re: Geode Performance how to push the cpu to 100%?
How much data are you pushing over the wire? If you are network-bound adding further TCP connections won’t help. You would need either a faster channel (e.g. 10g) or bonding across multiple 1g NICs.
Anthony
On Jun 9, 2017, at 10:55 PM, Xu, Nan <nx...@baml.com>> wrote:
Just an update, I assign more cpu to my server VM, also give more cpu to client. it’s not helpful, it’s means bigger box doesn’t help. I guess find out where is the bound is crucial now because all of our box in prod are big ones and we want to use them fully.
From: Michael Stolz [mailto:mstolz@pivotal.io]
Sent: Friday, June 09, 2017 7:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
You may need to use tools like top and sat and oostat to find out more
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771
On Jun 9, 2017 6:15 PM, "Xu, Nan" <nx...@baml.com>> wrote:
Agree that it is more IO bound but the only place I can think of is TCP/network, but adding more tcp connections don't help. Ideas?
Sent with BlackBerry Work (www.blackberry.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackberry.com&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4OtC5nIh8Nm1DFqN6UdqJIhmuqGBE-fxTVeMO_6MSfY&s=vjWLpMVDUpGJoP6pKya7sVCRWNKpfILWtmAqaqJrp_E&e=>)
________________________________
From: Michael Stolz <ms...@pivotal.io>>
Sent: Jun 9, 2017 5:05 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Sounds like you are becoming more and more I/O bound.
Remember, each put requires a complete round-trip including a layer-7 acknowledgement back to the caller to put(k,v).
So the longer each put takes the more I/O wait that thread has to do.
Have a look at iostat to see how to measure that.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771<tel:(631)%20835-4771>
On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <nx...@baml.com>> wrote:
So I further my testing and start to change message size.
The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client still send as quick as possible using 128 tcp connections. I can see with the message become bigger. My message send rate goes down, which is expected. But the cpu usage in the server side also goes down, the system become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%.
I run another program on the server, and able to use the unused cpus . so it is not the OS.
My socket buffer size is default, which is about 30k. checked my network between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s network bandwidth is not a problem.
I was hoping just by adding more tcp to client->server will make use all cpus and get better throughput but it not happen. Any thoughts?
I don’t want to tune the socket_buffer_size because it surely make one single tcp more efficient. But if I just open more tcp and I have the bandwidth and cpu, I should be able to get better performance too. is this assumption right?
Thanks,
Nan
From: Xu, Nan
Sent: Friday, June 09, 2017 1:55 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: RE: Geode Performance how to push the cpu to 100%?
Ya, after I finish my testing, I will do a write at the wiki.
Nan
From: Gregory Chase [mailto:gchase@pivotal.io]
Sent: Friday, June 09, 2017 12:59 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi Nan,
I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache Geode wiki<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com>> wrote:
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io<ma...@pivotal.io>]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--
Greg Chase
Product team
http://www.pivotal.io/big-data<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
Pivotal Software
http://www.pivotal.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
650-215-0477<tel:(650)%20215-0477>
@GregChase
Blog: http://geekmarketing.biz/<https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available athttp://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available athttp://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Anthony Baker <ab...@pivotal.io>.
How much data are you pushing over the wire? If you are network-bound adding further TCP connections won’t help. You would need either a faster channel (e.g. 10g) or bonding across multiple 1g NICs.
Anthony
> On Jun 9, 2017, at 10:55 PM, Xu, Nan <nx...@baml.com> wrote:
>
> Just an update, I assign more cpu to my server VM, also give more cpu to client. it’s not helpful, it’s means bigger box doesn’t help. I guess find out where is the bound is crucial now because all of our box in prod are big ones and we want to use them fully.
>
> From: Michael Stolz [mailto:mstolz@pivotal.io]
> Sent: Friday, June 09, 2017 7:32 PM
> To: user@geode.apache.org
> Subject: Re: Geode Performance how to push the cpu to 100%?
>
> You may need to use tools like top and sat and oostat to find out more
>
> --
> Mike Stolz
> Principal Engineer - Gemfire Product Manager
> Mobile: 631-835-4771
>
>
> On Jun 9, 2017 6:15 PM, "Xu, Nan" <nxu2@baml.com <ma...@baml.com>> wrote:
> Agree that it is more IO bound but the only place I can think of is TCP/network, but adding more tcp connections don't help. Ideas?
>
>
> Sent with BlackBerry Work (www.blackberry.com <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackberry.com&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4OtC5nIh8Nm1DFqN6UdqJIhmuqGBE-fxTVeMO_6MSfY&s=vjWLpMVDUpGJoP6pKya7sVCRWNKpfILWtmAqaqJrp_E&e=>)
> From: Michael Stolz <mstolz@pivotal.io <ma...@pivotal.io>>
> Sent: Jun 9, 2017 5:05 PM
> To: user@geode.apache.org <ma...@geode.apache.org>
> Subject: Re: Geode Performance how to push the cpu to 100%?
>
> Sounds like you are becoming more and more I/O bound.
>
> Remember, each put requires a complete round-trip including a layer-7 acknowledgement back to the caller to put(k,v).
>
> So the longer each put takes the more I/O wait that thread has to do.
>
> Have a look at iostat to see how to measure that.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771 <tel:(631)%20835-4771>
>
> On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <nxu2@baml.com <ma...@baml.com>> wrote:
> So I further my testing and start to change message size.
>
> The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client still send as quick as possible using 128 tcp connections. I can see with the message become bigger. My message send rate goes down, which is expected. But the cpu usage in the server side also goes down, the system become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%.
>
> I run another program on the server, and able to use the unused cpus . so it is not the OS.
>
> My socket buffer size is default, which is about 30k. checked my network between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s network bandwidth is not a problem.
>
> I was hoping just by adding more tcp to client->server will make use all cpus and get better throughput but it not happen. Any thoughts?
>
> I don’t want to tune the socket_buffer_size because it surely make one single tcp more efficient. But if I just open more tcp and I have the bandwidth and cpu, I should be able to get better performance too. is this assumption right?
>
> Thanks,
> Nan
>
> From: Xu, Nan
> Sent: Friday, June 09, 2017 1:55 PM
> To: user@geode.apache.org <ma...@geode.apache.org>
> Subject: RE: Geode Performance how to push the cpu to 100%?
>
> Ya, after I finish my testing, I will do a write at the wiki.
>
> Nan
>
> From: Gregory Chase [mailto:gchase@pivotal.io <ma...@pivotal.io>]
> Sent: Friday, June 09, 2017 12:59 PM
> To: user@geode.apache.org <ma...@geode.apache.org>
> Subject: Re: Geode Performance how to push the cpu to 100%?
>
> Hi Nan,
> I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
>
> If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
>
> The article will be useful anywhere, but if you need a place, the Apache Geode wiki <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
>
> Thanks in advance on behalf of future Geode users.
>
> -Greg
>
> On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nxu2@baml.com <ma...@baml.com>> wrote:
> Hi,
>
> Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
>
> Thanks,
> Nan
> From: Vincent Ford [mailto:vford@pivotal.io <ma...@pivotal.io>]
> Sent: Friday, June 09, 2017 12:32 PM
>
> To: user@geode.apache.org <ma...@geode.apache.org>
> Subject: Re: Geode Performance how to push the cpu to 100%?
>
> Hi,
>
> As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
>
> The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
>
> You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
>
> Vince Ford
> GemFire Toolsmith Engineering
> Beaverton, OR USA
> http://www.pivotal.io <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
> Open Source Project Geode https://geode.incubator.apache.org/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
>
> On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <mstolz@pivotal.io <ma...@pivotal.io>> wrote:
> Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
>
> # cat somebigfile | cat | cat... >/dev/null
>
> If that can't pin cpu then you know it's the operating system.
>
> --
> Mike Stolz
> Principal Engineer - Gemfire Product Manager
> Mobile: 631-835-4771 <tel:(631)%20835-4771>
>
> On Jun 9, 2017 9:22 AM, "Xu, Nan" <nxu2@baml.com <ma...@baml.com>> wrote:
> Thanks for pointing this out, very useful.
>
> I change the way I test the performance and get a result I can NOT explain.
>
> I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
>
> The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
>
> I can see there are 128 tcp connections between them and I send about 50,000 message/s
>
> Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
>
> Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
>
> Thanks,
> Nan
>
> From: Akihiro Kitada [mailto:akitada@pivotal.io <ma...@pivotal.io>]
> Sent: Thursday, June 08, 2017 7:13 PM
> To: user@geode.apache.org <ma...@geode.apache.org>
> Subject: Re: Geode Performance how to push the cpu to 100%?
>
> Hello Nan,
>
> Why don't you check Geode statistics
>
> http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
>
> Maybe, disk I/O or some other causes could be involved...
>
>
>
> --
> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736 <tel:+81%2080-3716-3736>
> Support.Pivotal.io <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <tel:(877)%20477-2269>
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
>
>
> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nxu2@baml.com <ma...@baml.com>>:
> Hi,
>
> I am trying to see the limit of geode performance.
>
> Here is what I did.
>
> Single machine: OS: centos 7. 8 core. 2.6G.
>
> Create a single locator and a single server and a single region.
>
> Only configuration is
> Server conserve-sockets= false
> Region is PARTITION
> Client and server running on the same machine.
>
>
> In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
>
> Why I cannot push it to 100%? I am suspecting
> 1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
>
> 2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
>
>
> Thanks,
> Nan
>
> Client side program.
>
> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
> .set("log-level", "info")
> .set("conserve-sockets", "false")
> .setPoolMinConnections(4)
> .setPoolMaxConnections(12)
> .setPoolThreadLocalConnections(true)
> .create
>
> val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
> val region1: Region[String, String] = regionFactory.create(region)
>
> implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
> var j = new AtomicLong(0)
> for (i <- 1 to 16) yield Future {
> while(true){
> val cj = j.addAndGet(1)
> region1.get("" + (rnd.nextInt(2000) + 1))
> }
> }
>
>
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
>
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
>
>
>
> --
> Greg Chase
>
> Product team
> http://www.pivotal.io/big-data <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
>
> Pivotal Software
> http://www.pivotal.io/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
>
> 650-215-0477 <tel:(650)%20215-0477>
> @GregChase
> Blog: http://geekmarketing.biz/ <https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available athttp://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available athttp://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer <http://www.bankofamerica.com/emaildisclaimer>. If you are not the intended recipient, please delete this message.
RE: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Just an update, I assign more cpu to my server VM, also give more cpu to client. it’s not helpful, it’s means bigger box doesn’t help. I guess find out where is the bound is crucial now because all of our box in prod are big ones and we want to use them fully.
From: Michael Stolz [mailto:mstolz@pivotal.io]
Sent: Friday, June 09, 2017 7:32 PM
To: user@geode.apache.org
Subject: Re: Geode Performance how to push the cpu to 100%?
You may need to use tools like top and sat and oostat to find out more
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771
On Jun 9, 2017 6:15 PM, "Xu, Nan" <nx...@baml.com>> wrote:
Agree that it is more IO bound but the only place I can think of is TCP/network, but adding more tcp connections don't help. Ideas?
Sent with BlackBerry Work (www.blackberry.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackberry.com&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=4OtC5nIh8Nm1DFqN6UdqJIhmuqGBE-fxTVeMO_6MSfY&s=vjWLpMVDUpGJoP6pKya7sVCRWNKpfILWtmAqaqJrp_E&e=>)
________________________________
From: Michael Stolz <ms...@pivotal.io>>
Sent: Jun 9, 2017 5:05 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Sounds like you are becoming more and more I/O bound.
Remember, each put requires a complete round-trip including a layer-7 acknowledgement back to the caller to put(k,v).
So the longer each put takes the more I/O wait that thread has to do.
Have a look at iostat to see how to measure that.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771<tel:(631)%20835-4771>
On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <nx...@baml.com>> wrote:
So I further my testing and start to change message size.
The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client still send as quick as possible using 128 tcp connections. I can see with the message become bigger. My message send rate goes down, which is expected. But the cpu usage in the server side also goes down, the system become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%.
I run another program on the server, and able to use the unused cpus . so it is not the OS.
My socket buffer size is default, which is about 30k. checked my network between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s network bandwidth is not a problem.
I was hoping just by adding more tcp to client->server will make use all cpus and get better throughput but it not happen. Any thoughts?
I don’t want to tune the socket_buffer_size because it surely make one single tcp more efficient. But if I just open more tcp and I have the bandwidth and cpu, I should be able to get better performance too. is this assumption right?
Thanks,
Nan
From: Xu, Nan
Sent: Friday, June 09, 2017 1:55 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: RE: Geode Performance how to push the cpu to 100%?
Ya, after I finish my testing, I will do a write at the wiki.
Nan
From: Gregory Chase [mailto:gchase@pivotal.io]
Sent: Friday, June 09, 2017 12:59 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi Nan,
I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache Geode wiki<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com>> wrote:
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io<ma...@pivotal.io>]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--
Greg Chase
Product team
http://www.pivotal.io/big-data<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
Pivotal Software
http://www.pivotal.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
650-215-0477<tel:(650)%20215-0477>
@GregChase
Blog: http://geekmarketing.biz/<https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Michael Stolz <ms...@pivotal.io>.
You may need to use tools like top and sat and oostat to find out more
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771
On Jun 9, 2017 6:15 PM, "Xu, Nan" <nx...@baml.com> wrote:
> Agree that it is more IO bound but the only place I can think of is
> TCP/network, but adding more tcp connections don't help. Ideas?
>
>
> Sent with BlackBerry Work (www.blackberry.com)
> ------------------------------
> *From: *Michael Stolz <ms...@pivotal.io>
> *Sent: *Jun 9, 2017 5:05 PM
> *To: *user@geode.apache.org
> *Subject: *Re: Geode Performance how to push the cpu to 100%?
>
> Sounds like you are becoming more and more I/O bound.
>
> Remember, each put requires a complete round-trip including a layer-7
> acknowledgement back to the caller to put(k,v).
>
> So the longer each put takes the more I/O wait that thread has to do.
>
> Have a look at iostat to see how to measure that.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Manager
> Mobile: +1-631-835-4771 <(631)%20835-4771>
>
> On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <nx...@baml.com> wrote:
>
>> So I further my testing and start to change message size.
>>
>>
>>
>> The payload from 10 bytes to 1k bytes then to 10k bytes then 100K.
>> Client still send as quick as possible using 128 tcp connections. I can see
>> with the message become bigger. My message send rate goes down, which is
>> expected. But the cpu usage in the server side also goes down, the system
>> become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at
>> 2%.
>>
>>
>>
>> I run another program on the server, and able to use the unused cpus . so
>> it is not the OS.
>>
>>
>>
>> My socket buffer size is default, which is about 30k. checked my network
>> between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s
>> network bandwidth is not a problem.
>>
>>
>>
>> I was hoping just by adding more tcp to client->server will make use all
>> cpus and get better throughput but it not happen. Any thoughts?
>>
>>
>>
>> I don’t want to tune the socket_buffer_size because it surely make one
>> single tcp more efficient. But if I just open more tcp and I have the
>> bandwidth and cpu, I should be able to get better performance too. is this
>> assumption right?
>>
>>
>>
>> Thanks,
>>
>> Nan
>>
>>
>>
>> *From:* Xu, Nan
>> *Sent:* Friday, June 09, 2017 1:55 PM
>> *To:* user@geode.apache.org
>> *Subject:* RE: Geode Performance how to push the cpu to 100%?
>>
>>
>>
>> Ya, after I finish my testing, I will do a write at the wiki.
>>
>>
>>
>> Nan
>>
>>
>>
>> *From:* Gregory Chase [mailto:gchase@pivotal.io <gc...@pivotal.io>]
>> *Sent:* Friday, June 09, 2017 12:59 PM
>> *To:* user@geode.apache.org
>> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>>
>>
>>
>> Hi Nan,
>> I think you started a very interesting discussion here that I other users
>> can benefit from - especially the advice you helped create.
>>
>>
>>
>> If your test case is general, I'm wondering if you would be willing to
>> contribute a short document explaining your throughput test, how you
>> achieved it, your results, and tips & tricks along the way.
>>
>>
>>
>> The article will be useful anywhere, but if you need a place, the Apache
>> Geode wiki
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=>
>> would be an excellent place.
>>
>>
>>
>> Thanks in advance on behalf of future Geode users.
>>
>>
>>
>> -Greg
>>
>>
>>
>> On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com> wrote:
>>
>> Hi,
>>
>>
>>
>> Thanks for all the suggestions. As Mike point out, this is because
>> of the operating system. I start another while(true) {i+=1} program on the
>> server and cannot make the cpu higher. To this point. I think it’s fair to
>> say it’s the operating system and in the current setup, I need to turn the
>> operating system to achieve better performance.
>>
>>
>>
>> Thanks,
>>
>> Nan
>>
>> *From:* Vincent Ford [mailto:vford@pivotal.io]
>> *Sent:* Friday, June 09, 2017 12:32 PM
>>
>>
>> *To:* user@geode.apache.org
>> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>>
>>
>>
>> Hi,
>>
>>
>>
>> As part of your test environment you should use two or more servers, as
>> the basic design of Geode is around distribution across a cluster of
>> servers. In a test it is best to use distribution with at least two servers
>> as your client will be able to distribute it's load across the two
>> different members and although in a single host test setup you may not see
>> significant initial improvements but as you scale for production loads
>> across multiple machines you will get performance and stability benefits
>> from having tested with distribution from the start.
>>
>>
>>
>> The conserve sockets parameter effects the way servers communicate
>> between themselves by opening additional sockets between the peer/server
>> nodes for handling updates, so setting this parameter doesn't have as much
>> impact with a single server test.
>>
>>
>>
>> You may need to do virtualization and OS tuning to get the full impact,
>> increasing the socket buffer sizes should help if you haven't done this
>> already...
>>
>>
>> *Vince Ford*
>>
>> GemFire Toolsmith Engineering
>>
>> Beaverton, OR USA
>>
>> http://www.pivotal.io
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
>>
>> Open Source Project Geode https://geode.incubator.apache.org/
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
>>
>>
>>
>> On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io> wrote:
>>
>> Try doing just raw pipe communications between a bunch of processes to
>> see if you can pin the cpu.
>>
>> # cat somebigfile | cat | cat... >/dev/null
>>
>> If that can't pin cpu then you know it's the operating system.
>>
>> --
>> Mike Stolz
>> Principal Engineer - Gemfire Product Manager
>> Mobile: 631-835-4771 <(631)%20835-4771>
>>
>>
>>
>> On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com> wrote:
>>
>> Thanks for pointing this out, very useful.
>>
>>
>>
>> I change the way I test the performance and get a result I can NOT
>> explain.
>>
>>
>>
>> I use 2 separate virtual machine. 1 run client, 1 run server, both
>> siting on the same physical box.
>>
>>
>>
>> The client put small message (about 10 bytes) as quick as possible
>> through 128 threads. Both client and server have the conserve_socket=false
>>
>>
>>
>> I can see there are 128 tcp connections between them and I send about
>> 50,000 message/s
>>
>>
>>
>> Server have 4 core and 3 out of 4 is constantly 100%, but one core is
>> only 30%. On the server, I only run 1 locator and 1 server and no other
>> program. Region is PARTITION. I publish about 2000 keys.
>>
>>
>>
>> Why there is a core only 30%. My Point is, if I can use the last core
>> more, I might able to publisher even quicker.
>>
>>
>>
>> Thanks,
>>
>> Nan
>>
>>
>>
>> *From:* Akihiro Kitada [mailto:akitada@pivotal.io]
>> *Sent:* Thursday, June 08, 2017 7:13 PM
>> *To:* user@geode.apache.org
>> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>>
>>
>>
>> Hello Nan,
>>
>>
>>
>> Why don't you check Geode statistics
>>
>>
>>
>> http://geode.apache.org/docs/guide/11/reference/statistics/s
>> tatistics_list.html
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
>>
>>
>>
>> Maybe, disk I/O or some other causes could be involved...
>>
>>
>>
>>
>>
>>
>> --
>>
>> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
>> <+81%2080-3716-3736>
>> Support.Pivotal.io
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=>
>> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269>
>> [image: support]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=>
>> [image: twitter]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=>
>> [image: linkedin]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=>
>> [image: facebook]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=>
>> [image: google plus]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=>
>> [image: youtube]
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
>>
>>
>>
>>
>>
>> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
>>
>> Hi,
>>
>>
>>
>> I am trying to see the limit of geode performance.
>>
>>
>>
>> Here is what I did.
>>
>>
>>
>> Single machine: OS: centos 7. 8 core. 2.6G.
>>
>>
>>
>> Create a single locator and a single server and a single region.
>>
>>
>>
>> Only configuration is
>>
>> Server conserve-sockets= false
>>
>> Region is PARTITION
>>
>> Client and server running on the same machine.
>>
>>
>>
>>
>>
>> In my client. I setup a 16 thread pool to get data. But I can only push
>> the cpu to around 90—93% on the centos.
>>
>>
>>
>> Why I cannot push it to 100%? I am suspecting
>>
>> 1. The tcp connection between the server and client is not fast
>> enough. Maybe increase the number of tcp connection? I only see one
>> connection between client and server.
>>
>> 2. There are some lock at the server? I realize that I can push
>> the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
>> so maybe there are some other setting I am missing.
>>
>>
>>
>> Thanks,
>>
>> Nan
>>
>>
>>
>> Client side program.
>>
>>
>>
>> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
>> 10334)
>>
>> .set("log-level", "info")
>>
>> .set("conserve-sockets", "false")
>>
>> .setPoolMinConnections(4)
>>
>> .setPoolMaxConnections(12)
>>
>> .setPoolThreadLocalConnections(true)
>>
>> .create
>>
>>
>>
>> val regionFactory: ClientRegionFactory[String, String] =
>> cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
>>
>> val region1: Region[String, String] = regionFactory.create(region)
>>
>>
>>
>> implicit val ec = ExecutionContext.fromExecutorS
>> ervice(Executors.newFixedThreadPool(16))
>>
>> var j = new AtomicLong(0)
>>
>> for (i <- 1 to 16) yield Future {
>>
>> while(true){
>>
>> val cj = j.addAndGet(1)
>>
>> region1.get("" + (rnd.nextInt(2000) + 1))
>>
>> }
>>
>> }
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.bankofamerica.com/emaildisclaimer. If you are not the
>> intended recipient, please delete this message.
>>
>>
>> ------------------------------
>>
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.bankofamerica.com/emaildisclaimer. If you are not the
>> intended recipient, please delete this message.
>>
>>
>>
>>
>> ------------------------------
>>
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.bankofamerica.com/emaildisclaimer. If you are not the
>> intended recipient, please delete this message.
>>
>>
>>
>>
>>
>> --
>>
>> Greg Chase
>>
>>
>>
>> Product team
>>
>> http://www.pivotal.io/big-data
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
>>
>>
>>
>> Pivotal Software
>>
>> http://www.pivotal.io/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
>>
>>
>>
>> 650-215-0477 <(650)%20215-0477>
>>
>> @GregChase
>>
>> Blog: http://geekmarketing.biz/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
>>
>>
>> ------------------------------
>>
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.bankofamerica.com/emaildisclaimer. If you are not the
>> intended recipient, please delete this message.
>> ------------------------------
>> This message, and any attachments, is for the intended recipient(s) only,
>> may contain information that is privileged, confidential and/or proprietary
>> and subject to important terms and conditions available at
>> http://www.bankofamerica.com/emaildisclaimer. If you are not the
>> intended recipient, please delete this message.
>>
>
> ------------------------------
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
Re: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Agree that it is more IO bound but the only place I can think of is TCP/network, but adding more tcp connections don't help. Ideas?
Sent with BlackBerry Work (www.blackberry.com)
________________________________
From: Michael Stolz <ms...@pivotal.io>
Sent: Jun 9, 2017 5:05 PM
To: user@geode.apache.org
Subject: Re: Geode Performance how to push the cpu to 100%?
Sounds like you are becoming more and more I/O bound.
Remember, each put requires a complete round-trip including a layer-7 acknowledgement back to the caller to put(k,v).
So the longer each put takes the more I/O wait that thread has to do.
Have a look at iostat to see how to measure that.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771
On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <nx...@baml.com>> wrote:
So I further my testing and start to change message size.
The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client still send as quick as possible using 128 tcp connections. I can see with the message become bigger. My message send rate goes down, which is expected. But the cpu usage in the server side also goes down, the system become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%.
I run another program on the server, and able to use the unused cpus . so it is not the OS.
My socket buffer size is default, which is about 30k. checked my network between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s network bandwidth is not a problem.
I was hoping just by adding more tcp to client->server will make use all cpus and get better throughput but it not happen. Any thoughts?
I don’t want to tune the socket_buffer_size because it surely make one single tcp more efficient. But if I just open more tcp and I have the bandwidth and cpu, I should be able to get better performance too. is this assumption right?
Thanks,
Nan
From: Xu, Nan
Sent: Friday, June 09, 2017 1:55 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: RE: Geode Performance how to push the cpu to 100%?
Ya, after I finish my testing, I will do a write at the wiki.
Nan
From: Gregory Chase [mailto:gchase@pivotal.io]
Sent: Friday, June 09, 2017 12:59 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi Nan,
I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache Geode wiki<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com>> wrote:
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io<ma...@pivotal.io>]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--
Greg Chase
Product team
http://www.pivotal.io/big-data<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
Pivotal Software
http://www.pivotal.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
650-215-0477<tel:(650)%20215-0477>
@GregChase
Blog: http://geekmarketing.biz/<https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Michael Stolz <ms...@pivotal.io>.
Sounds like you are becoming more and more I/O bound.
Remember, each put requires a complete round-trip including a layer-7
acknowledgement back to the caller to put(k,v).
So the longer each put takes the more I/O wait that thread has to do.
Have a look at iostat to see how to measure that.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771
On Fri, Jun 9, 2017 at 2:27 PM, Xu, Nan <nx...@baml.com> wrote:
> So I further my testing and start to change message size.
>
>
>
> The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client
> still send as quick as possible using 128 tcp connections. I can see with
> the message become bigger. My message send rate goes down, which is
> expected. But the cpu usage in the server side also goes down, the system
> become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at
> 2%.
>
>
>
> I run another program on the server, and able to use the unused cpus . so
> it is not the OS.
>
>
>
> My socket buffer size is default, which is about 30k. checked my network
> between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s
> network bandwidth is not a problem.
>
>
>
> I was hoping just by adding more tcp to client->server will make use all
> cpus and get better throughput but it not happen. Any thoughts?
>
>
>
> I don’t want to tune the socket_buffer_size because it surely make one
> single tcp more efficient. But if I just open more tcp and I have the
> bandwidth and cpu, I should be able to get better performance too. is this
> assumption right?
>
>
>
> Thanks,
>
> Nan
>
>
>
> *From:* Xu, Nan
> *Sent:* Friday, June 09, 2017 1:55 PM
> *To:* user@geode.apache.org
> *Subject:* RE: Geode Performance how to push the cpu to 100%?
>
>
>
> Ya, after I finish my testing, I will do a write at the wiki.
>
>
>
> Nan
>
>
>
> *From:* Gregory Chase [mailto:gchase@pivotal.io <gc...@pivotal.io>]
> *Sent:* Friday, June 09, 2017 12:59 PM
> *To:* user@geode.apache.org
> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>
>
>
> Hi Nan,
> I think you started a very interesting discussion here that I other users
> can benefit from - especially the advice you helped create.
>
>
>
> If your test case is general, I'm wondering if you would be willing to
> contribute a short document explaining your throughput test, how you
> achieved it, your results, and tips & tricks along the way.
>
>
>
> The article will be useful anywhere, but if you need a place, the Apache
> Geode wiki
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=>
> would be an excellent place.
>
>
>
> Thanks in advance on behalf of future Geode users.
>
>
>
> -Greg
>
>
>
> On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com> wrote:
>
> Hi,
>
>
>
> Thanks for all the suggestions. As Mike point out, this is because of
> the operating system. I start another while(true) {i+=1} program on the
> server and cannot make the cpu higher. To this point. I think it’s fair to
> say it’s the operating system and in the current setup, I need to turn the
> operating system to achieve better performance.
>
>
>
> Thanks,
>
> Nan
>
> *From:* Vincent Ford [mailto:vford@pivotal.io]
> *Sent:* Friday, June 09, 2017 12:32 PM
>
>
> *To:* user@geode.apache.org
> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>
>
>
> Hi,
>
>
>
> As part of your test environment you should use two or more servers, as
> the basic design of Geode is around distribution across a cluster of
> servers. In a test it is best to use distribution with at least two servers
> as your client will be able to distribute it's load across the two
> different members and although in a single host test setup you may not see
> significant initial improvements but as you scale for production loads
> across multiple machines you will get performance and stability benefits
> from having tested with distribution from the start.
>
>
>
> The conserve sockets parameter effects the way servers communicate
> between themselves by opening additional sockets between the peer/server
> nodes for handling updates, so setting this parameter doesn't have as much
> impact with a single server test.
>
>
>
> You may need to do virtualization and OS tuning to get the full impact,
> increasing the socket buffer sizes should help if you haven't done this
> already...
>
>
> *Vince Ford*
>
> GemFire Toolsmith Engineering
>
> Beaverton, OR USA
>
> http://www.pivotal.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
>
> Open Source Project Geode https://geode.incubator.apache.org/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
>
>
>
> On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io> wrote:
>
> Try doing just raw pipe communications between a bunch of processes to see
> if you can pin the cpu.
>
> # cat somebigfile | cat | cat... >/dev/null
>
> If that can't pin cpu then you know it's the operating system.
>
> --
> Mike Stolz
> Principal Engineer - Gemfire Product Manager
> Mobile: 631-835-4771 <(631)%20835-4771>
>
>
>
> On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com> wrote:
>
> Thanks for pointing this out, very useful.
>
>
>
> I change the way I test the performance and get a result I can NOT explain.
>
>
>
> I use 2 separate virtual machine. 1 run client, 1 run server, both siting
> on the same physical box.
>
>
>
> The client put small message (about 10 bytes) as quick as possible through
> 128 threads. Both client and server have the conserve_socket=false
>
>
>
> I can see there are 128 tcp connections between them and I send about
> 50,000 message/s
>
>
>
> Server have 4 core and 3 out of 4 is constantly 100%, but one core is only
> 30%. On the server, I only run 1 locator and 1 server and no other
> program. Region is PARTITION. I publish about 2000 keys.
>
>
>
> Why there is a core only 30%. My Point is, if I can use the last core
> more, I might able to publisher even quicker.
>
>
>
> Thanks,
>
> Nan
>
>
>
> *From:* Akihiro Kitada [mailto:akitada@pivotal.io]
> *Sent:* Thursday, June 08, 2017 7:13 PM
> *To:* user@geode.apache.org
> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>
>
>
> Hello Nan,
>
>
>
> Why don't you check Geode statistics
>
>
>
> http://geode.apache.org/docs/guide/11/reference/statistics/
> statistics_list.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
>
>
>
> Maybe, disk I/O or some other causes could be involved...
>
>
>
>
>
>
> --
>
> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
> <+81%2080-3716-3736>
> Support.Pivotal.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=>
> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269>
> [image: support]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=>
> [image: twitter]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=>
> [image: linkedin]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=>
> [image: facebook]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=>
> [image: google plus]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=>
> [image: youtube]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
>
>
>
>
>
> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
>
> Hi,
>
>
>
> I am trying to see the limit of geode performance.
>
>
>
> Here is what I did.
>
>
>
> Single machine: OS: centos 7. 8 core. 2.6G.
>
>
>
> Create a single locator and a single server and a single region.
>
>
>
> Only configuration is
>
> Server conserve-sockets= false
>
> Region is PARTITION
>
> Client and server running on the same machine.
>
>
>
>
>
> In my client. I setup a 16 thread pool to get data. But I can only push
> the cpu to around 90—93% on the centos.
>
>
>
> Why I cannot push it to 100%? I am suspecting
>
> 1. The tcp connection between the server and client is not fast
> enough. Maybe increase the number of tcp connection? I only see one
> connection between client and server.
>
> 2. There are some lock at the server? I realize that I can push
> the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
> so maybe there are some other setting I am missing.
>
>
>
> Thanks,
>
> Nan
>
>
>
> Client side program.
>
>
>
> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
> 10334)
>
> .set("log-level", "info")
>
> .set("conserve-sockets", "false")
>
> .setPoolMinConnections(4)
>
> .setPoolMaxConnections(12)
>
> .setPoolThreadLocalConnections(true)
>
> .create
>
>
>
> val regionFactory: ClientRegionFactory[String, String] = cache.
> createClientRegionFactory(ClientRegionShortcut.PROXY)
>
> val region1: Region[String, String] = regionFactory.create(region)
>
>
>
> implicit val ec = ExecutionContext.fromExecutorService(Executors.
> newFixedThreadPool(16))
>
> var j = new AtomicLong(0)
>
> for (i <- 1 to 16) yield Future {
>
> while(true){
>
> val cj = j.addAndGet(1)
>
> region1.get("" + (rnd.nextInt(2000) + 1))
>
> }
>
> }
>
>
>
>
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
>
>
>
> --
>
> Greg Chase
>
>
>
> Product team
>
> http://www.pivotal.io/big-data
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
>
>
>
> Pivotal Software
>
> http://www.pivotal.io/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
>
>
>
> 650-215-0477 <(650)%20215-0477>
>
> @GregChase
>
> Blog: http://geekmarketing.biz/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
> ------------------------------
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
RE: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Btw, the region is partition so no disk involved if my understanding is right. The communication is happening between VMs sitting on one physical box.
From: Xu, Nan
Sent: Friday, June 09, 2017 4:28 PM
To: user@geode.apache.org
Subject: RE: Geode Performance how to push the cpu to 100%?
So I further my testing and start to change message size.
The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client still send as quick as possible using 128 tcp connections. I can see with the message become bigger. My message send rate goes down, which is expected. But the cpu usage in the server side also goes down, the system become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%.
I run another program on the server, and able to use the unused cpus . so it is not the OS.
My socket buffer size is default, which is about 30k. checked my network between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s network bandwidth is not a problem.
I was hoping just by adding more tcp to client->server will make use all cpus and get better throughput but it not happen. Any thoughts?
I don’t want to tune the socket_buffer_size because it surely make one single tcp more efficient. But if I just open more tcp and I have the bandwidth and cpu, I should be able to get better performance too. is this assumption right?
Thanks,
Nan
From: Xu, Nan
Sent: Friday, June 09, 2017 1:55 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: RE: Geode Performance how to push the cpu to 100%?
Ya, after I finish my testing, I will do a write at the wiki.
Nan
From: Gregory Chase [mailto:gchase@pivotal.io]
Sent: Friday, June 09, 2017 12:59 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi Nan,
I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache Geode wiki<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com>> wrote:
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io<ma...@pivotal.io>]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--
Greg Chase
Product team
http://www.pivotal.io/big-data<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
Pivotal Software
http://www.pivotal.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
650-215-0477
@GregChase
Blog: http://geekmarketing.biz/<https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
RE: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
So I further my testing and start to change message size.
The payload from 10 bytes to 1k bytes then to 10k bytes then 100K. Client still send as quick as possible using 128 tcp connections. I can see with the message become bigger. My message send rate goes down, which is expected. But the cpu usage in the server side also goes down, the system become not 100% cpu but around 80% at 100k for 3 cpus, with 1 cpu only at 2%.
I run another program on the server, and able to use the unused cpus . so it is not the OS.
My socket buffer size is default, which is about 30k. checked my network between the VM, it’s 3.2G/s. the peak transfer rate is about 170M so it’s network bandwidth is not a problem.
I was hoping just by adding more tcp to client->server will make use all cpus and get better throughput but it not happen. Any thoughts?
I don’t want to tune the socket_buffer_size because it surely make one single tcp more efficient. But if I just open more tcp and I have the bandwidth and cpu, I should be able to get better performance too. is this assumption right?
Thanks,
Nan
From: Xu, Nan
Sent: Friday, June 09, 2017 1:55 PM
To: user@geode.apache.org
Subject: RE: Geode Performance how to push the cpu to 100%?
Ya, after I finish my testing, I will do a write at the wiki.
Nan
From: Gregory Chase [mailto:gchase@pivotal.io]
Sent: Friday, June 09, 2017 12:59 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi Nan,
I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache Geode wiki<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com>> wrote:
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io<ma...@pivotal.io>]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--
Greg Chase
Product team
http://www.pivotal.io/big-data<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
Pivotal Software
http://www.pivotal.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
650-215-0477
@GregChase
Blog: http://geekmarketing.biz/<https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
RE: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Ya, after I finish my testing, I will do a write at the wiki.
Nan
From: Gregory Chase [mailto:gchase@pivotal.io]
Sent: Friday, June 09, 2017 12:59 PM
To: user@geode.apache.org
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi Nan,
I think you started a very interesting discussion here that I other users can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to contribute a short document explaining your throughput test, how you achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache Geode wiki<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_GEODE_Index&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bl3a12MFHzrMqYOfmMZfrc7vHsjxENp_9KPU6zR6i0&e=> would be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com>> wrote:
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io<ma...@pivotal.io>]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
--
Greg Chase
Product team
http://www.pivotal.io/big-data<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_big-2Ddata&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=VLy5cokKkGdjuiMsxf3idMSh0jML9nqpTsHS6-MXX_Y&e=>
Pivotal Software
http://www.pivotal.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=0bhu3pLH6ph60-XLNsbBe5mNHcVKc7Jeq8LTqq95ZCk&e=>
650-215-0477
@GregChase
Blog: http://geekmarketing.biz/<https://urldefense.proofpoint.com/v2/url?u=http-3A__geekmarketing.biz_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=rNHj7BI4Xsux7CTdYDB0q0A1ov_d9Zk-FA4byeuK6bE&s=6-faGR28Kdu12PGCOo-yWwq-v35HZSoWhiIMeTai47o&e=>
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Gregory Chase <gc...@pivotal.io>.
Hi Nan,
I think you started a very interesting discussion here that I other users
can benefit from - especially the advice you helped create.
If your test case is general, I'm wondering if you would be willing to
contribute a short document explaining your throughput test, how you
achieved it, your results, and tips & tricks along the way.
The article will be useful anywhere, but if you need a place, the Apache
Geode wiki <https://cwiki.apache.org/confluence/display/GEODE/Index> would
be an excellent place.
Thanks in advance on behalf of future Geode users.
-Greg
On Fri, Jun 9, 2017 at 10:51 AM, Xu, Nan <nx...@baml.com> wrote:
> Hi,
>
>
>
> Thanks for all the suggestions. As Mike point out, this is because of
> the operating system. I start another while(true) {i+=1} program on the
> server and cannot make the cpu higher. To this point. I think it’s fair to
> say it’s the operating system and in the current setup, I need to turn the
> operating system to achieve better performance.
>
>
>
> Thanks,
>
> Nan
>
> *From:* Vincent Ford [mailto:vford@pivotal.io]
> *Sent:* Friday, June 09, 2017 12:32 PM
>
> *To:* user@geode.apache.org
> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>
>
>
> Hi,
>
>
>
> As part of your test environment you should use two or more servers, as
> the basic design of Geode is around distribution across a cluster of
> servers. In a test it is best to use distribution with at least two servers
> as your client will be able to distribute it's load across the two
> different members and although in a single host test setup you may not see
> significant initial improvements but as you scale for production loads
> across multiple machines you will get performance and stability benefits
> from having tested with distribution from the start.
>
>
>
> The conserve sockets parameter effects the way servers communicate
> between themselves by opening additional sockets between the peer/server
> nodes for handling updates, so setting this parameter doesn't have as much
> impact with a single server test.
>
>
>
> You may need to do virtualization and OS tuning to get the full impact,
> increasing the socket buffer sizes should help if you haven't done this
> already...
>
>
> *Vince Ford*
>
> GemFire Toolsmith Engineering
>
> Beaverton, OR USA
>
> http://www.pivotal.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
>
> Open Source Project Geode https://geode.incubator.apache.org/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
>
>
>
> On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io> wrote:
>
> Try doing just raw pipe communications between a bunch of processes to see
> if you can pin the cpu.
>
> # cat somebigfile | cat | cat... >/dev/null
>
> If that can't pin cpu then you know it's the operating system.
>
> --
> Mike Stolz
> Principal Engineer - Gemfire Product Manager
> Mobile: 631-835-4771 <(631)%20835-4771>
>
>
>
> On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com> wrote:
>
> Thanks for pointing this out, very useful.
>
>
>
> I change the way I test the performance and get a result I can NOT explain.
>
>
>
> I use 2 separate virtual machine. 1 run client, 1 run server, both siting
> on the same physical box.
>
>
>
> The client put small message (about 10 bytes) as quick as possible through
> 128 threads. Both client and server have the conserve_socket=false
>
>
>
> I can see there are 128 tcp connections between them and I send about
> 50,000 message/s
>
>
>
> Server have 4 core and 3 out of 4 is constantly 100%, but one core is only
> 30%. On the server, I only run 1 locator and 1 server and no other
> program. Region is PARTITION. I publish about 2000 keys.
>
>
>
> Why there is a core only 30%. My Point is, if I can use the last core
> more, I might able to publisher even quicker.
>
>
>
> Thanks,
>
> Nan
>
>
>
> *From:* Akihiro Kitada [mailto:akitada@pivotal.io]
> *Sent:* Thursday, June 08, 2017 7:13 PM
> *To:* user@geode.apache.org
> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>
>
>
> Hello Nan,
>
>
>
> Why don't you check Geode statistics
>
>
>
> http://geode.apache.org/docs/guide/11/reference/statistics/
> statistics_list.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
>
>
>
> Maybe, disk I/O or some other causes could be involved...
>
>
>
>
>
>
> --
>
> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
> <+81%2080-3716-3736>
> Support.Pivotal.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=>
> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269>
> [image: support]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=>
> [image: twitter]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=>
> [image: linkedin]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=>
> [image: facebook]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=>
> [image: google plus]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=>
> [image: youtube]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
>
>
>
>
>
> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
>
> Hi,
>
>
>
> I am trying to see the limit of geode performance.
>
>
>
> Here is what I did.
>
>
>
> Single machine: OS: centos 7. 8 core. 2.6G.
>
>
>
> Create a single locator and a single server and a single region.
>
>
>
> Only configuration is
>
> Server conserve-sockets= false
>
> Region is PARTITION
>
> Client and server running on the same machine.
>
>
>
>
>
> In my client. I setup a 16 thread pool to get data. But I can only push
> the cpu to around 90—93% on the centos.
>
>
>
> Why I cannot push it to 100%? I am suspecting
>
> 1. The tcp connection between the server and client is not fast
> enough. Maybe increase the number of tcp connection? I only see one
> connection between client and server.
>
> 2. There are some lock at the server? I realize that I can push
> the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
> so maybe there are some other setting I am missing.
>
>
>
> Thanks,
>
> Nan
>
>
>
> Client side program.
>
>
>
> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
> 10334)
>
> .set("log-level", "info")
>
> .set("conserve-sockets", "false")
>
> .setPoolMinConnections(4)
>
> .setPoolMaxConnections(12)
>
> .setPoolThreadLocalConnections(true)
>
> .create
>
>
>
> val regionFactory: ClientRegionFactory[String, String] = cache.
> createClientRegionFactory(ClientRegionShortcut.PROXY)
>
> val region1: Region[String, String] = regionFactory.create(region)
>
>
>
> implicit val ec = ExecutionContext.fromExecutorService(Executors.
> newFixedThreadPool(16))
>
> var j = new AtomicLong(0)
>
> for (i <- 1 to 16) yield Future {
>
> while(true){
>
> val cj = j.addAndGet(1)
>
> region1.get("" + (rnd.nextInt(2000) + 1))
>
> }
>
> }
>
>
>
>
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
>
>
> ------------------------------
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
--
Greg Chase
Product team
http://www.pivotal.io/big-data
Pivotal Software
http://www.pivotal.io/
650-215-0477
@GregChase
Blog: http://geekmarketing.biz/
RE: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Hi,
Thanks for all the suggestions. As Mike point out, this is because of the operating system. I start another while(true) {i+=1} program on the server and cannot make the cpu higher. To this point. I think it’s fair to say it’s the operating system and in the current setup, I need to turn the operating system to achieve better performance.
Thanks,
Nan
From: Vincent Ford [mailto:vford@pivotal.io]
Sent: Friday, June 09, 2017 12:32 PM
To: user@geode.apache.org
Subject: Re: Geode Performance how to push the cpu to 100%?
Hi,
As part of your test environment you should use two or more servers, as the basic design of Geode is around distribution across a cluster of servers. In a test it is best to use distribution with at least two servers as your client will be able to distribute it's load across the two different members and although in a single host test setup you may not see significant initial improvements but as you scale for production loads across multiple machines you will get performance and stability benefits from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between themselves by opening additional sockets between the peer/server nodes for handling updates, so setting this parameter doesn't have as much impact with a single server test.
You may need to do virtualization and OS tuning to get the full impact, increasing the socket buffer sizes should help if you haven't done this already...
Vince Ford
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pivotal.io&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=7432wySy951E1aDXgYyglk2Xj8rgshw_R9vTwr1WRZI&e=>
Open Source Project Geode https://geode.incubator.apache.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__network.pivotal.io_products_project-2Dgeode&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=jA6gcP4ARbtU41FCBhxbyMYIc6RtwsaQ---Me6CGLDY&s=KGH_HqslLBGi5D7BkuOLerMbLuUJS5NEwOscVbG7l-s&e=>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io>> wrote:
Try doing just raw pipe communications between a bunch of processes to see if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771<tel:(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com>> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io<ma...@pivotal.io>]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org<ma...@geode.apache.org>
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736<tel:+81%2080-3716-3736>
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269<tel:(877)%20477-2269>
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Vincent Ford <vf...@pivotal.io>.
Hi,
As part of your test environment you should use two or more servers, as the
basic design of Geode is around distribution across a cluster of servers.
In a test it is best to use distribution with at least two servers as your
client will be able to distribute it's load across the two different
members and although in a single host test setup you may not see
significant initial improvements but as you scale for production loads
across multiple machines you will get performance and stability benefits
from having tested with distribution from the start.
The conserve sockets parameter effects the way servers communicate between
themselves by opening additional sockets between the peer/server nodes for
handling updates, so setting this parameter doesn't have as much impact
with a single server test.
You may need to do virtualization and OS tuning to get the full impact,
increasing the socket buffer sizes should help if you haven't done this
already...
*Vince Ford*
GemFire Toolsmith Engineering
Beaverton, OR USA
http://www.pivotal.io
Open Source Project Geode https://geode.incubator.apache.org/
<https://network.pivotal.io/products/project-geode>
On Fri, Jun 9, 2017 at 9:37 AM, Michael Stolz <ms...@pivotal.io> wrote:
> Try doing just raw pipe communications between a bunch of processes to see
> if you can pin the cpu.
>
> # cat somebigfile | cat | cat... >/dev/null
>
> If that can't pin cpu then you know it's the operating system.
>
>
> --
> Mike Stolz
> Principal Engineer - Gemfire Product Manager
> Mobile: 631-835-4771 <(631)%20835-4771>
>
> On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com> wrote:
>
> Thanks for pointing this out, very useful.
>
>
>
> I change the way I test the performance and get a result I can NOT explain.
>
>
>
> I use 2 separate virtual machine. 1 run client, 1 run server, both siting
> on the same physical box.
>
>
>
> The client put small message (about 10 bytes) as quick as possible through
> 128 threads. Both client and server have the conserve_socket=false
>
>
>
> I can see there are 128 tcp connections between them and I send about
> 50,000 message/s
>
>
>
> Server have 4 core and 3 out of 4 is constantly 100%, but one core is only
> 30%. On the server, I only run 1 locator and 1 server and no other
> program. Region is PARTITION. I publish about 2000 keys.
>
>
>
> Why there is a core only 30%. My Point is, if I can use the last core
> more, I might able to publisher even quicker.
>
>
>
> Thanks,
>
> Nan
>
>
>
> *From:* Akihiro Kitada [mailto:akitada@pivotal.io]
> *Sent:* Thursday, June 08, 2017 7:13 PM
> *To:* user@geode.apache.org
> *Subject:* Re: Geode Performance how to push the cpu to 100%?
>
>
>
> Hello Nan,
>
>
>
> Why don't you check Geode statistics
>
>
>
> http://geode.apache.org/docs/guide/11/reference/statistics/s
> tatistics_list.html
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
>
>
>
> Maybe, disk I/O or some other causes could be involved...
>
>
>
>
>
>
> --
>
> Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
> <+81%2080-3716-3736>
> Support.Pivotal.io
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=>
> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269>
> [image: support]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=>
> [image: twitter]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=>
> [image: linkedin]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=>
> [image: facebook]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=>
> [image: google plus]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=>
> [image: youtube]
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
>
>
>
>
>
> 2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
>
> Hi,
>
>
>
> I am trying to see the limit of geode performance.
>
>
>
> Here is what I did.
>
>
>
> Single machine: OS: centos 7. 8 core. 2.6G.
>
>
>
> Create a single locator and a single server and a single region.
>
>
>
> Only configuration is
>
> Server conserve-sockets= false
>
> Region is PARTITION
>
> Client and server running on the same machine.
>
>
>
>
>
> In my client. I setup a 16 thread pool to get data. But I can only push
> the cpu to around 90—93% on the centos.
>
>
>
> Why I cannot push it to 100%? I am suspecting
>
> 1. The tcp connection between the server and client is not fast
> enough. Maybe increase the number of tcp connection? I only see one
> connection between client and server.
>
> 2. There are some lock at the server? I realize that I can push
> the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
> so maybe there are some other setting I am missing.
>
>
>
> Thanks,
>
> Nan
>
>
>
> Client side program.
>
>
>
> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
> 10334)
>
> .set("log-level", "info")
>
> .set("conserve-sockets", "false")
>
> .setPoolMinConnections(4)
>
> .setPoolMaxConnections(12)
>
> .setPoolThreadLocalConnections(true)
>
> .create
>
>
>
> val regionFactory: ClientRegionFactory[String, String] =
> cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
>
> val region1: Region[String, String] = regionFactory.create(region)
>
>
>
> implicit val ec = ExecutionContext.fromExecutorS
> ervice(Executors.newFixedThreadPool(16))
>
> var j = new AtomicLong(0)
>
> for (i <- 1 to 16) yield Future {
>
> while(true){
>
> val cj = j.addAndGet(1)
>
> region1.get("" + (rnd.nextInt(2000) + 1))
>
> }
>
> }
>
>
>
>
>
>
> ------------------------------
>
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
> ------------------------------
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>
>
>
RE: Geode Performance how to push the cpu to 100%?
Posted by Michael Stolz <ms...@pivotal.io>.
Try doing just raw pipe communications between a bunch of processes to see
if you can pin the cpu.
# cat somebigfile | cat | cat... >/dev/null
If that can't pin cpu then you know it's the operating system.
--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771 <(631)%20835-4771>
On Jun 9, 2017 9:22 AM, "Xu, Nan" <nx...@baml.com> wrote:
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting
on the same physical box.
The client put small message (about 10 bytes) as quick as possible through
128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about
50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only
30%. On the server, I only run 1 locator and 1 server and no other
program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more,
I might able to publisher even quicker.
Thanks,
Nan
*From:* Akihiro Kitada [mailto:akitada@pivotal.io]
*Sent:* Thursday, June 08, 2017 7:13 PM
*To:* user@geode.apache.org
*Subject:* Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/s
tatistics_list.html
<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
<+81%2080-3716-3736>
Support.Pivotal.io
<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=>
| Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269 <(877)%20477-2269>
[image: support]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=>
[image: twitter]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=>
[image: linkedin]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=>
[image: facebook]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=>
[image: google plus]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=>
[image: youtube]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push
the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast
enough. Maybe increase the number of tcp connection? I only see one
connection between client and server.
2. There are some lock at the server? I realize that I can push the
cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] =
cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorS
ervice(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
------------------------------
This message, and any attachments, is for the intended recipient(s) only,
may contain information that is privileged, confidential and/or proprietary
and subject to important terms and conditions available at
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
recipient, please delete this message.
------------------------------
This message, and any attachments, is for the intended recipient(s) only,
may contain information that is privileged, confidential and/or proprietary
and subject to important terms and conditions available at
http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
recipient, please delete this message.
RE: Geode Performance how to push the cpu to 100%?
Posted by "Xu, Nan" <nx...@baml.com>.
Thanks for pointing this out, very useful.
I change the way I test the performance and get a result I can NOT explain.
I use 2 separate virtual machine. 1 run client, 1 run server, both siting on the same physical box.
The client put small message (about 10 bytes) as quick as possible through 128 threads. Both client and server have the conserve_socket=false
I can see there are 128 tcp connections between them and I send about 50,000 message/s
Server have 4 core and 3 out of 4 is constantly 100%, but one core is only 30%. On the server, I only run 1 locator and 1 server and no other program. Region is PARTITION. I publish about 2000 keys.
Why there is a core only 30%. My Point is, if I can use the last core more, I might able to publisher even quicker.
Thanks,
Nan
From: Akihiro Kitada [mailto:akitada@pivotal.io]
Sent: Thursday, June 08, 2017 7:13 PM
To: user@geode.apache.org
Subject: Re: Geode Performance how to push the cpu to 100%?
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__geode.apache.org_docs_guide_11_reference_statistics_statistics-5Flist.html&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=A8tHBTyifwAGQtQwFPmi2z75-p7wg9WcJAOGSENZvyc&e=>
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
Support.Pivotal.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=KlPLE6HUp_oymyeU0phGa7yGv1y0AI9a37IzLaxgqvE&e=> | Mon-Fri 9:00am to 5:30pm JST | 1-877-477-2269
[support]<https://urldefense.proofpoint.com/v2/url?u=https-3A__support.pivotal.io_&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=D8zlpynD1sHe0VhZffs1DV9myytvY4nswTaVL_QeIfA&e=> [twitter] <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_pivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=MxtrJ8TClgbIiOLjX1LfZWiX5DvIqKSm98HD2q_5Vt4&e=> [linkedin] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_3048967&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=jLQapJvEDrWcoj9wbQJv6BfF8POND-BdcjO8OcltqZU&e=> [facebook] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_pivotalsoftware&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=40iw0ItXxQdUmXtC1EwHZDQd9YSUcAJmmxseZJGByIk&e=> [google plus] <https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_-2BPivotal&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=kr2r2DNQWN972jvd5YSJV6Zbsr1FBUtPS-cHUOLF-Eg&e=> [youtube] <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.youtube.com_playlist-3Flist-3DPLAdzTan-5FeSPScpj2J50ErtzR9ANSzv3kl&d=DwMFaQ&c=SFszdw3oxIkTvaP4xmzq_apLU3uL-3SxdAPNkldf__Q&r=HB5LZowSGF4DiMmOUsCX6Q&m=pJzOQaxj-UphjOo5W_YGcoqC10MXjGdGZhv2XJV5O74&s=X3efwi-zDaQQNnXPfr6HjHdoUQp-esDmJhY1t_XU8LM&e=>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>>:
Hi,
I am trying to see the limit of geode performance.
Here is what I did.
Single machine: OS: centos 7. 8 core. 2.6G.
Create a single locator and a single server and a single region.
Only configuration is
Server conserve-sockets= false
Region is PARTITION
Client and server running on the same machine.
In my client. I setup a 16 thread pool to get data. But I can only push the cpu to around 90—93% on the centos.
Why I cannot push it to 100%? I am suspecting
1. The tcp connection between the server and client is not fast enough. Maybe increase the number of tcp connection? I only see one connection between client and server.
2. There are some lock at the server? I realize that I can push the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true), so maybe there are some other setting I am missing.
Thanks,
Nan
Client side program.
val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host, 10334)
.set("log-level", "info")
.set("conserve-sockets", "false")
.setPoolMinConnections(4)
.setPoolMaxConnections(12)
.setPoolThreadLocalConnections(true)
.create
val regionFactory: ClientRegionFactory[String, String] = cache.createClientRegionFactory(ClientRegionShortcut.PROXY)
val region1: Region[String, String] = regionFactory.create(region)
implicit val ec = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(16))
var j = new AtomicLong(0)
for (i <- 1 to 16) yield Future {
while(true){
val cj = j.addAndGet(1)
region1.get("" + (rnd.nextInt(2000) + 1))
}
}
________________________________
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: Geode Performance how to push the cpu to 100%?
Posted by Akihiro Kitada <ak...@pivotal.io>.
Hello Nan,
Why don't you check Geode statistics
http://geode.apache.org/docs/guide/11/reference/statistics/statistics_list.html
Maybe, disk I/O or some other causes could be involved...
--
Akihiro Kitada | Staff Customer Engineer | +81 80 3716 3736
Support.Pivotal.io <http://support.pivotal.io/> | Mon-Fri 9:00am to
5:30pm JST | 1-877-477-2269
[image: support] <https://support.pivotal.io/> [image: twitter]
<https://twitter.com/pivotal> [image: linkedin]
<https://www.linkedin.com/company/3048967> [image: facebook]
<https://www.facebook.com/pivotalsoftware> [image: google plus]
<https://plus.google.com/+Pivotal> [image: youtube]
<https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl>
2017-06-09 6:09 GMT+09:00 Xu, Nan <nx...@baml.com>:
> Hi,
>
>
>
> I am trying to see the limit of geode performance.
>
>
>
> Here is what I did.
>
>
>
> Single machine: OS: centos 7. 8 core. 2.6G.
>
>
>
> Create a single locator and a single server and a single region.
>
>
>
> Only configuration is
>
> Server conserve-sockets= false
>
> Region is PARTITION
>
> Client and server running on the same machine.
>
>
>
>
>
> In my client. I setup a 16 thread pool to get data. But I can only push
> the cpu to around 90—93% on the centos.
>
>
>
> Why I cannot push it to 100%? I am suspecting
>
> 1. The tcp connection between the server and client is not fast
> enough. Maybe increase the number of tcp connection? I only see one
> connection between client and server.
>
> 2. There are some lock at the server? I realize that I can push
> the cpu from 50% to 90 by just adding the setPoolThreadLocalConnections(true),
> so maybe there are some other setting I am missing.
>
>
>
> Thanks,
>
> Nan
>
>
>
> Client side program.
>
>
>
> val cache: ClientCache = new ClientCacheFactory().addPoolLocator(host,
> 10334)
>
> .set("log-level", "info")
>
> .set("conserve-sockets", "false")
>
> .setPoolMinConnections(4)
>
> .setPoolMaxConnections(12)
>
> .setPoolThreadLocalConnections(true)
>
> .create
>
>
>
> val regionFactory: ClientRegionFactory[String, String] = cache.
> createClientRegionFactory(ClientRegionShortcut.PROXY)
>
> val region1: Region[String, String] = regionFactory.create(region)
>
>
>
> implicit val ec = ExecutionContext.fromExecutorService(Executors.
> newFixedThreadPool(16))
>
> var j = new AtomicLong(0)
>
> for (i <- 1 to 16) yield Future {
>
> while(true){
>
> val cj = j.addAndGet(1)
>
> region1.get("" + (rnd.nextInt(2000) + 1))
>
> }
>
> }
>
>
>
>
>
>
>
> ------------------------------
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the intended
> recipient, please delete this message.
>