You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Di Li <di...@apple.com> on 2016/12/13 08:45:12 UTC
benchmark ATS
Hey Guys,
When I doing some benchmark for outbound proxy, and has http_cache enabled, well, first of all, the performance are pretty low, I guess I didn’t do it right with the cache enabled, 2nd when I use wrk to have 512 connection with 40 thread to go through proxy with http, it cause a core dump, here’s the trace
And when I disable the http.cache, the performance has went up a lot, and no more coredump at all.
FATAL: CacheRead.cc:249: failed assert `w->alternate.valid()`
traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
traffic_server: Aborted (Signal sent by tkill() 20136 1001)
traffic_server - STACK TRACE:
/ngs/app/oproxy/trafficserver/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0xc3)[0x50730d]
/lib64/libc.so.6(+0x35670)[0x2aaaad284670]
/lib64/libc.so.6(gsignal+0x37)[0x2aaaad2845f7]
/lib64/libc.so.6(abort+0x148)[0x2aaaad285ce8]
/ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z12ink_fatal_vaPKcP13__va_list_tag+0x0)[0x2aaaaad0cffd]
/ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z9ink_fatalPKcz+0x0)[0x2aaaaad0d0b4]
/ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z10ink_pfatalPKcz+0x0)[0x2aaaaad0d179]
/ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(+0x3be7a)[0x2aaaaad0ae7a]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7CacheVC20openReadChooseWriterEiP5Event+0x3be)[0x74292c]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7CacheVC18openReadFromWriterEiP5Event+0x390)[0x743032]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x5b0)[0x7422aa]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationPK12HttpCacheKeybP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xbf)[0x728a83]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readERK12HttpCacheKey+0xf7)[0x5c86b5]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x118)[0x5c884a]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x276)[0x5e6956]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xc3a)[0x5f11e6]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x1a2)[0x5f074e]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1c0)[0x5da444]
/ngs/app/oproxy/trafficserver/bin/traffic_server(TSHttpTxnReenable+0x198)[0x52b339]
/ngs/app/oproxy/trafficserver/lib/trafficserver/plugins/stats_over_http.so(+0x1c86)[0x2aadfef71c86]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xe8)[0x51eff6]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7APIHook6invokeEiPv+0x73)[0x51f9ab]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x671)[0x5dab31]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x124b)[0x5d88d9]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x270)[0x5df19c]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0xe9)[0x5d7667]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x5e)[0x5daf70]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_add_to_listEiPv+0x27e)[0x5d6c86]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP22ProxyClientTransactionP14IOBufferReader+0x484)[0x5d7506]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN22ProxyClientTransaction15new_transactionEv+0x2d0)[0x552138]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18Http1ClientSession15new_transactionEv+0xc0)[0x5cabfe]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18Http1ClientSession16state_keep_aliveEiPv+0x1d3)[0x5ca715]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
/ngs/app/oproxy/trafficserver/bin/traffic_server[0x78973b]
/ngs/app/oproxy/trafficserver/bin/traffic_server[0x78a494]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18UnixNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x2b)[0x78c5ef]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x6f0)[0x7816fc]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x136)[0x7ac1ec]
/ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x4ae)[0x7ac818]
/ngs/app/oproxy/trafficserver/bin/traffic_server[0x7ab784]
/lib64/libpthread.so.0(+0x7dc5)[0x2aaaad615dc5]
/lib64/libc.so.6(clone+0x6d)[0x2aaaad345c4d]
Thanks,
Di Li
Re: benchmark ATS
Posted by James Peach <jo...@gmail.com>.
> On Dec 13, 2016, at 12:45 AM, Di Li <di...@apple.com> wrote:
>
> Hey Guys,
>
> When I doing some benchmark for outbound proxy, and has http_cache enabled, well, first of all, the performance are pretty low, I guess I didn’t do it right with the cache enabled, 2nd when I use wrk to have 512 connection with 40 thread to go through proxy with http, it cause a core dump, here’s the trace
>
> And when I disable the http.cache, the performance has went up a lot, and no more coredump at all.
>
>
> FATAL: CacheRead.cc:249: failed assert `w->alternate.valid()`
> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
> traffic_server - STACK TRACE:
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0xc3)[0x50730d]
> /lib64/libc.so.6(+0x35670)[0x2aaaad284670]
> /lib64/libc.so.6(gsignal+0x37)[0x2aaaad2845f7]
> /lib64/libc.so.6(abort+0x148)[0x2aaaad285ce8]
> /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z12ink_fatal_vaPKcP13__va_list_tag+0x0)[0x2aaaaad0cffd]
This is an assert, there should be a corresponding message in logs somewhere.
> /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z9ink_fatalPKcz+0x0)[0x2aaaaad0d0b4]
> /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(_Z10ink_pfatalPKcz+0x0)[0x2aaaaad0d179]
> /ngs/app/oproxy/trafficserver/lib/trafficserver/libtsutil.so.6(+0x3be7a)[0x2aaaaad0ae7a]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7CacheVC20openReadChooseWriterEiP5Event+0x3be)[0x74292c]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7CacheVC18openReadFromWriterEiP5Event+0x390)[0x743032]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x5b0)[0x7422aa]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN14CacheProcessor9open_readEP12ContinuationPK12HttpCacheKeybP7HTTPHdrP21CacheLookupHttpConfigl13CacheFragType+0xbf)[0x728a83]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN11HttpCacheSM18do_cache_open_readERK12HttpCacheKey+0xf7)[0x5c86b5]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0x118)[0x5c884a]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x276)[0x5e6956]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xc3a)[0x5f11e6]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x1a2)[0x5f074e]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0xfd)[0x5db00f]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x1c0)[0x5da444]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(TSHttpTxnReenable+0x198)[0x52b339]
> /ngs/app/oproxy/trafficserver/lib/trafficserver/plugins/stats_over_http.so(+0x1c86)[0x2aadfef71c86]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0xe8)[0x51eff6]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7APIHook6invokeEiPv+0x73)[0x51f9ab]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x671)[0x5dab31]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x5e)[0x5f060a]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x1ae)[0x5f05a4]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x124b)[0x5d88d9]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0x270)[0x5df19c]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM32setup_client_read_request_headerEv+0xe9)[0x5d7667]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x5e)[0x5daf70]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x95c)[0x5dae1c]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM23do_api_callout_internalEv+0x1ca)[0x5e8f00]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM14do_api_calloutEv+0x27)[0x5f7979]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM17state_add_to_listEiPv+0x27e)[0x5d6c86]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP22ProxyClientTransactionP14IOBufferReader+0x484)[0x5d7506]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN22ProxyClientTransaction15new_transactionEv+0x2d0)[0x552138]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18Http1ClientSession15new_transactionEv+0xc0)[0x5cabfe]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18Http1ClientSession16state_keep_aliveEiPv+0x1d3)[0x5ca715]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
> /ngs/app/oproxy/trafficserver/bin/traffic_server[0x78973b]
> /ngs/app/oproxy/trafficserver/bin/traffic_server[0x78a494]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN18UnixNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x2b)[0x78c5ef]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x6f0)[0x7816fc]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN12Continuation11handleEventEiPv+0x6c)[0x50a34e]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x136)[0x7ac1ec]
> /ngs/app/oproxy/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x4ae)[0x7ac818]
> /ngs/app/oproxy/trafficserver/bin/traffic_server[0x7ab784]
> /lib64/libpthread.so.0(+0x7dc5)[0x2aaaad615dc5]
> /lib64/libc.so.6(clone+0x6d)[0x2aaaad345c4d]
>
>
> Thanks,
> Di Li
>
>
>
>
>
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
doesn’t seem I can get more than 50000 req/s with ab or with wrk
i have cache disabled, and I’m only running 1G link right now, so far with WRK or AB, the real destination is a simple nginx server running on http only, and so far I can’t get more than 50000 req/s
I will dig more on it, but is there something obvious that need to be turned on the proxy server side ?
./wrk -c 256 -t 256 -d 10 -s ./scripts/via_proxy_get1.lua http://x.x.x.x:8080
Running 10s test @ http://x.x.x.x:8080
256 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.63ms 35.20ms 777.98ms 97.53%
Req/Sec 169.26 55.34 530.00 74.19%
429181 requests in 10.10s, 354.04MB read
Requests/sec: 42492.09
Transfer/sec: 35.05MB
Benchmarking 10.12.17.58 [through :8080] (be patient)
Completed 1000000 requests
Completed 2000000 requests
Completed 3000000 requests
^C
Server Software: ATS/6.2.0
Server Hostname: 10.12.17.58
Server Port: 80
Document Path: /
Document Length: 612 bytes
Concurrency Level: 500
Time taken for tests: 88.196 seconds
Complete requests: 3674058
Failed requests: 0
Write errors: 0
Keep-Alive requests: 3674058
Total transferred: 3156015822 bytes
HTML transferred: 2248523496 bytes
Requests per second: 41657.91 [#/sec] (mean)
Time per request: 12.003 [ms] (mean)
Time per request: 0.024 [ms] (mean, across all concurrent requests)
Transfer rate: 34945.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.5 0 64
Processing: 1 12 10.9 7 687
Waiting: 1 12 10.9 7 687
Total: 1 12 11.0 7 702
Percentage of the requests served within a certain time (ms)
Thanks,
Di Li
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ SHIELD :: Self-Service Load-Balancer (LB) and Web App Firewall (WAF)
+ http://shield.apple.com <http://shield.apple.com/>
+
+ SHIELD classes:
+ http://shield.apple.com/FAQ/doku.php?id=start#shield_classes_trainings_tutorials <http://shield.apple.com/FAQ/doku.php?id=start#shield_classes_trainings_tutorials>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> On Dec 13, 2016, at 10:20 AM, Di Li <di...@apple.com> wrote:
>
> Hey Steve,
>
> Can you share some details on config or performance turning or results ?
>
>
> Thanks,
> Di Li
>
>
>
>
>> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>
>> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
>> I used Apache Bench for this.
>>
>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>> <image001.png>
>>
>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Date: Tuesday, December 13, 2016 at 12:32 PM
>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Subject: Re: benchmark ATS
>>
>> using 6.2.0, repeatable
>>
>>
>> Thanks,
>> Di Li
>>
>>
>>
>>
>> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>>
>>
>> Am 13.12.2016 um 09:45 schrieb Di Li:
>>
>> When I doing some benchmark for outbound proxy, and has http_cache
>> enabled, well, first of all, the performance are pretty low, I guess I
>> didn’t do it right with the cache enabled, 2nd when I use wrk to have
>> 512 connection with 40 thread to go through proxy with http, it cause a
>> core dump, here’s the trace
>>
>> And when I disable the http.cache, the performance has went up a lot,
>> and no more coredump at all.
>>
>>
>> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
>> `w->alternate.valid()`
>> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
>> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
>> traffic_server - STACK TRACE
>>
>> is this repeatable?
>> which version of ATS?
>>
>> at least mention the software version should be common-sense
>>
>> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>>
>
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
Hey Steve,
It really depends on which piece you want to test it out, we have a feature which will work on every connection, so that’s the reason we trying to get a baseline request per sec performance before we enable the feature and then try different ways to implement that feature to see which one scales out and makes the most sense.
If your use case is to serve big post data, and bandwidth is your primary concern then you have test it right
Thanks,
Di Li
> On Dec 14, 2016, at 4:36 PM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Im interested to see what the specs of your machine are and the numbers you are getting and how you set up your tests…
>
> Benchmarking is something few people really do- I spent a bunch of time and was pretty happy with the results but would love to see some peers work…
>
>
>
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
> Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Date: Wednesday, December 14, 2016 at 4:39 PM
> To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Subject: Re: benchmark ATS
>
> Hi Yongming,
>
>
> Thanks for list, we are not using cache part, and that’s being disabled from our end, and our goal is to test the req/s , instead of the bandwidth it will use, most the thing we already tested out.
>
> I will check what the IRQ balance, NIC drivers stuff as the rest already been tested.
>
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 14, 2016, at 10:06 AM, Yongming Zhao <ming.zym@gmail.com <ma...@gmail.com>> wrote:
>
> the cache performance may concern:
> 1, hit or miss
> 2, hit on disk or ram
> 3, how many concurrent connections
> 4, how many uniq objects
> 5, how many request per connections(keep alive)
> 6, is you binary a debug building?
> 7, are you debug tag set?
> 8, is disk io issue?
> …
>
> but you get 100% cpu, that means you reach the ATS limit, you may:
> 1, lower down the query/response size.
> 2, more keep-alive
> 3, try to tweak on the connections
> 4, take a look upon the cache & hits
> 5, try to make queries less collision in memory or dist(incase of the complex hit/miss/update situation)
> 6, try to evalue how many NET threads works better
> 7, try to figure out which thread make trouble
> 8, try to figure out how to lower down the SI cpu usage
> ...
>
> take a look of the jtest & httpload in the tree, a good test tool always better than anything else.
>
> 50000/qps definitely not a good mark :D
>
> - Yongming Zhao 赵永明
>
> 在 2016年12月14日，上午6:14，Di Li <di_li@apple.com <ma...@apple.com>> 写道：
>
> Hi Steve,
>
> Thanks for that info, I’m trying to use the small packets, each of request are just 200 byes, and responds in 900 bytes.
>
> I don’t worry too much about the bandwidth, the request per second is my primary testing object, I only use 1 server as client, 1 server as proxy, 1 server as destination, all dedicated boxes
>
> So far, I can’t pass 50000 req/s , I can see our proxy box has burned to 100% cpu usage with 24 cores. I’m still investigating if anything wrong on the config or the sysctl or IRQ balance
>
>
> Thanks,
> Di Li
>
>
>
>
>
> On Dec 13, 2016, at 2:01 PM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> Are you using multiple VMs to post via a single VM with ATS to a tuned HTTPD?
>
> I got upwards of 50K connections and 9gbps. I posted giant binary files as well… my concern isn’t connections per second its bandwidth throttling which doesn’t seem to happen with these massive posts.
>
> In today’s age of VMs if we need more hits/sec we’d just spin up more VMs with ATS.
>
>
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Date: Tuesday, December 13, 2016 at 4:44 PM
> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Subject: Re: benchmark ATS
>
> Hi Steve,
>
> We had those things before I made the post, and so far I stuck with the same num.
>
> Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
>
> https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html <https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html>
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 11:08 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> We use Ubuntu Server and in the end the only tuning was:
>
> · /etc/sysctl.conf
> o net.ipv4.tcp_tw_recycle = 1
> o net.core.somaxconn = 65535
> o net.ipv4.tcp_fin_timeout = 15
> o net.ipv4.tcp_keepalive_time = 300
> o net.ipv4.tcp_keepalive_probes = 5
> o net.ipv4.tcp_keepalive_intvl = 15
>
> But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
>
> The test was simply:
>
> ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache <http://[apache> traffic server forward proxy IP]:8080/index.html
>
> where post.txt is the file to post.
>
> You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
>
> To see the performance, we used commands ss-s and tops.
> Run these on all the machine involved to keep an eye on everything.
>
> This was all run manually and quickly.
>
> -Steve
>
>
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Date: Tuesday, December 13, 2016 at 1:20 PM
> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Subject: Re: benchmark ATS
>
> Hey Steve,
>
> Can you share some details on config or performance turning or results ?
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
> I used Apache Bench for this.
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Date: Tuesday, December 13, 2016 at 12:32 PM
> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Subject: Re: benchmark ATS
>
> using 6.2.0, repeatable
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>
>
> Am 13.12.2016 um 09:45 schrieb Di Li:
>
>
>
>
> When I doing some benchmark for outbound proxy, and has http_cache
> enabled, well, first of all, the performance are pretty low, I guess I
> didn’t do it right with the cache enabled, 2nd when I use wrk to have
> 512 connection with 40 thread to go through proxy with http, it cause a
> core dump, here’s the trace
>
> And when I disable the http.cache, the performance has went up a lot,
> and no more coredump at all.
>
>
> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
> `w->alternate.valid()`
> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
> traffic_server - STACK TRACE
>
> is this repeatable?
> which version of ATS?
>
> at least mention the software version should be common-sense
>
> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>
>
>
>
>
>
Re: benchmark ATS
Posted by "Lerner, Steve" <sl...@ebay.com>.
Im interested to see what the specs of your machine are and the numbers you are getting and how you set up your tests…
Benchmarking is something few people really do- I spent a bunch of time and was pretty happy with the results but would love to see some peers work…
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
[cid:image001.png@01D25641.606C29D0]
From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Date: Wednesday, December 14, 2016 at 4:39 PM
To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Subject: Re: benchmark ATS
Hi Yongming,
Thanks for list, we are not using cache part, and that’s being disabled from our end, and our goal is to test the req/s , instead of the bandwidth it will use, most the thing we already tested out.
I will check what the IRQ balance, NIC drivers stuff as the rest already been tested.
Thanks,
Di Li
On Dec 14, 2016, at 10:06 AM, Yongming Zhao <mi...@gmail.com>> wrote:
the cache performance may concern:
1, hit or miss
2, hit on disk or ram
3, how many concurrent connections
4, how many uniq objects
5, how many request per connections(keep alive)
6, is you binary a debug building?
7, are you debug tag set?
8, is disk io issue?
…
but you get 100% cpu, that means you reach the ATS limit, you may:
1, lower down the query/response size.
2, more keep-alive
3, try to tweak on the connections
4, take a look upon the cache & hits
5, try to make queries less collision in memory or dist(incase of the complex hit/miss/update situation)
6, try to evalue how many NET threads works better
7, try to figure out which thread make trouble
8, try to figure out how to lower down the SI cpu usage
...
take a look of the jtest & httpload in the tree, a good test tool always better than anything else.
50000/qps definitely not a good mark :D
- Yongming Zhao 赵永明
在 2016年12月14日，上午6:14，Di Li <di...@apple.com>> 写道：
Hi Steve,
Thanks for that info, I’m trying to use the small packets, each of request are just 200 byes, and responds in 900 bytes.
I don’t worry too much about the bandwidth, the request per second is my primary testing object, I only use 1 server as client, 1 server as proxy, 1 server as destination, all dedicated boxes
So far, I can’t pass 50000 req/s , I can see our proxy box has burned to 100% cpu usage with 24 cores. I’m still investigating if anything wrong on the config or the sysctl or IRQ balance
Thanks,
Di Li
On Dec 13, 2016, at 2:01 PM, Lerner, Steve <sl...@ebay.com>> wrote:
Are you using multiple VMs to post via a single VM with ATS to a tuned HTTPD?
I got upwards of 50K connections and 9gbps. I posted giant binary files as well… my concern isn’t connections per second its bandwidth throttling which doesn’t seem to happen with these massive posts.
In today’s age of VMs if we need more hits/sec we’d just spin up more VMs with ATS.
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
<image001.png>
From: <di...@apple.com>> on behalf of Di Li <di...@apple.com>>
Reply-To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Date: Tuesday, December 13, 2016 at 4:44 PM
To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Subject: Re: benchmark ATS
Hi Steve,
We had those things before I made the post, and so far I stuck with the same num.
Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html
Thanks,
Di Li
On Dec 13, 2016, at 11:08 AM, Lerner, Steve <sl...@ebay.com>> wrote:
We use Ubuntu Server and in the end the only tuning was:
• /etc/sysctl.conf
o net.ipv4.tcp_tw_recycle = 1
o net.core.somaxconn = 65535
o net.ipv4.tcp_fin_timeout = 15
o net.ipv4.tcp_keepalive_time = 300
o net.ipv4.tcp_keepalive_probes = 5
o net.ipv4.tcp_keepalive_intvl = 15
But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
The test was simply:
ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache traffic server forward proxy IP]:8080/index.html
where post.txt is the file to post.
You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
To see the performance, we used commands ss-s and tops.
Run these on all the machine involved to keep an eye on everything.
This was all run manually and quickly.
-Steve
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
<image001.png>
From: <di...@apple.com>> on behalf of Di Li <di...@apple.com>>
Reply-To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Date: Tuesday, December 13, 2016 at 1:20 PM
To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Subject: Re: benchmark ATS
Hey Steve,
Can you share some details on config or performance turning or results ?
Thanks,
Di Li
On Dec 13, 2016, at 9:46 AM, Lerner, Steve <sl...@ebay.com>> wrote:
I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
I used Apache Bench for this.
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
<image001.png>
From: <di...@apple.com>> on behalf of Di Li <di...@apple.com>>
Reply-To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Date: Tuesday, December 13, 2016 at 12:32 PM
To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Subject: Re: benchmark ATS
using 6.2.0, repeatable
Thanks,
Di Li
On Dec 13, 2016, at 1:28 AM, Reindl Harald <h....@thelounge.net>> wrote:
Am 13.12.2016 um 09:45 schrieb Di Li:
When I doing some benchmark for outbound proxy, and has http_cache
enabled, well, first of all, the performance are pretty low, I guess I
didn’t do it right with the cache enabled, 2nd when I use wrk to have
512 connection with 40 thread to go through proxy with http, it cause a
core dump, here’s the trace
And when I disable the http.cache, the performance has went up a lot,
and no more coredump at all.
FATAL: CacheRead.cc<http://cacheread.cc/> <http://cacheread.cc<http://cacheread.cc/>>:249: failed assert
`w->alternate.valid()`
traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
traffic_server: Aborted (Signal sent by tkill() 20136 1001)
traffic_server - STACK TRACE
is this repeatable?
which version of ATS?
at least mention the software version should be common-sense
had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
Hi Yongming,
Thanks for list, we are not using cache part, and that’s being disabled from our end, and our goal is to test the req/s , instead of the bandwidth it will use, most the thing we already tested out.
I will check what the IRQ balance, NIC drivers stuff as the rest already been tested.
Thanks,
Di Li
> On Dec 14, 2016, at 10:06 AM, Yongming Zhao <mi...@gmail.com> wrote:
>
> the cache performance may concern:
> 1, hit or miss
> 2, hit on disk or ram
> 3, how many concurrent connections
> 4, how many uniq objects
> 5, how many request per connections(keep alive)
> 6, is you binary a debug building?
> 7, are you debug tag set?
> 8, is disk io issue?
> …
>
> but you get 100% cpu, that means you reach the ATS limit, you may:
> 1, lower down the query/response size.
> 2, more keep-alive
> 3, try to tweak on the connections
> 4, take a look upon the cache & hits
> 5, try to make queries less collision in memory or dist(incase of the complex hit/miss/update situation)
> 6, try to evalue how many NET threads works better
> 7, try to figure out which thread make trouble
> 8, try to figure out how to lower down the SI cpu usage
> ...
>
> take a look of the jtest & httpload in the tree, a good test tool always better than anything else.
>
> 50000/qps definitely not a good mark :D
>
> - Yongming Zhao 赵永明
>
>> 在 2016年12月14日，上午6:14，Di Li <di_li@apple.com <ma...@apple.com>> 写道：
>>
>> Hi Steve,
>>
>> Thanks for that info, I’m trying to use the small packets, each of request are just 200 byes, and responds in 900 bytes.
>>
>> I don’t worry too much about the bandwidth, the request per second is my primary testing object, I only use 1 server as client, 1 server as proxy, 1 server as destination, all dedicated boxes
>>
>> So far, I can’t pass 50000 req/s , I can see our proxy box has burned to 100% cpu usage with 24 cores. I’m still investigating if anything wrong on the config or the sysctl or IRQ balance
>>
>>
>> Thanks,
>> Di Li
>>
>>
>>
>>
>>
>>> On Dec 13, 2016, at 2:01 PM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>>
>>> Are you using multiple VMs to post via a single VM with ATS to a tuned HTTPD?
>>>
>>> I got upwards of 50K connections and 9gbps. I posted giant binary files as well… my concern isn’t connections per second its bandwidth throttling which doesn’t seem to happen with these massive posts.
>>>
>>> In today’s age of VMs if we need more hits/sec we’d just spin up more VMs with ATS.
>>>
>>>
>>>
>>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>>> <image001.png>
>>>
>>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>>> Date: Tuesday, December 13, 2016 at 4:44 PM
>>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>>> Subject: Re: benchmark ATS
>>>
>>> Hi Steve,
>>>
>>> We had those things before I made the post, and so far I stuck with the same num.
>>>
>>> Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
>>>
>>> https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html <https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html>
>>>
>>>
>>> Thanks,
>>> Di Li
>>>
>>>
>>>
>>>
>>> On Dec 13, 2016, at 11:08 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>>
>>> We use Ubuntu Server and in the end the only tuning was:
>>>
>>> · /etc/sysctl.conf
>>> o net.ipv4.tcp_tw_recycle = 1
>>> o net.core.somaxconn = 65535
>>> o net.ipv4.tcp_fin_timeout = 15
>>> o net.ipv4.tcp_keepalive_time = 300
>>> o net.ipv4.tcp_keepalive_probes = 5
>>> o net.ipv4.tcp_keepalive_intvl = 15
>>>
>>> But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
>>>
>>> The test was simply:
>>>
>>> ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache <http://[apache> traffic server forward proxy IP]:8080/index.html
>>>
>>> where post.txt is the file to post.
>>>
>>> You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
>>>
>>> To see the performance, we used commands ss-s and tops.
>>> Run these on all the machine involved to keep an eye on everything.
>>>
>>> This was all run manually and quickly.
>>>
>>> -Steve
>>>
>>>
>>>
>>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>>> <image001.png>
>>>
>>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>>> Date: Tuesday, December 13, 2016 at 1:20 PM
>>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>>> Subject: Re: benchmark ATS
>>>
>>> Hey Steve,
>>>
>>> Can you share some details on config or performance turning or results ?
>>>
>>>
>>> Thanks,
>>> Di Li
>>>
>>>
>>>
>>>
>>> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>>
>>> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
>>> I used Apache Bench for this.
>>>
>>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>>> <image001.png>
>>>
>>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>>> Date: Tuesday, December 13, 2016 at 12:32 PM
>>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>>> Subject: Re: benchmark ATS
>>>
>>> using 6.2.0, repeatable
>>>
>>>
>>> Thanks,
>>> Di Li
>>>
>>>
>>>
>>>
>>> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>>>
>>>
>>> Am 13.12.2016 um 09:45 schrieb Di Li:
>>>
>>>
>>>
>>> When I doing some benchmark for outbound proxy, and has http_cache
>>> enabled, well, first of all, the performance are pretty low, I guess I
>>> didn’t do it right with the cache enabled, 2nd when I use wrk to have
>>> 512 connection with 40 thread to go through proxy with http, it cause a
>>> core dump, here’s the trace
>>>
>>> And when I disable the http.cache, the performance has went up a lot,
>>> and no more coredump at all.
>>>
>>>
>>> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
>>> `w->alternate.valid()`
>>> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
>>> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
>>> traffic_server - STACK TRACE
>>>
>>> is this repeatable?
>>> which version of ATS?
>>>
>>> at least mention the software version should be common-sense
>>>
>>> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>>>
>>>
>>>
>>
>
Re: benchmark ATS
Posted by Yongming Zhao <mi...@gmail.com>.
the cache performance may concern:
1, hit or miss
2, hit on disk or ram
3, how many concurrent connections
4, how many uniq objects
5, how many request per connections(keep alive)
6, is you binary a debug building?
7, are you debug tag set?
8, is disk io issue?
…
but you get 100% cpu, that means you reach the ATS limit, you may:
1, lower down the query/response size.
2, more keep-alive
3, try to tweak on the connections
4, take a look upon the cache & hits
5, try to make queries less collision in memory or dist(incase of the complex hit/miss/update situation)
6, try to evalue how many NET threads works better
7, try to figure out which thread make trouble
8, try to figure out how to lower down the SI cpu usage
...
take a look of the jtest & httpload in the tree, a good test tool always better than anything else.
50000/qps definitely not a good mark :D
- Yongming Zhao 赵永明
> 在 2016年12月14日，上午6:14，Di Li <di...@apple.com> 写道：
>
> Hi Steve,
>
> Thanks for that info, I’m trying to use the small packets, each of request are just 200 byes, and responds in 900 bytes.
>
> I don’t worry too much about the bandwidth, the request per second is my primary testing object, I only use 1 server as client, 1 server as proxy, 1 server as destination, all dedicated boxes
>
> So far, I can’t pass 50000 req/s , I can see our proxy box has burned to 100% cpu usage with 24 cores. I’m still investigating if anything wrong on the config or the sysctl or IRQ balance
>
>
> Thanks,
> Di Li
>
>
>
>
>
>> On Dec 13, 2016, at 2:01 PM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>
>> Are you using multiple VMs to post via a single VM with ATS to a tuned HTTPD?
>>
>> I got upwards of 50K connections and 9gbps. I posted giant binary files as well… my concern isn’t connections per second its bandwidth throttling which doesn’t seem to happen with these massive posts.
>>
>> In today’s age of VMs if we need more hits/sec we’d just spin up more VMs with ATS.
>>
>>
>>
>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>> <image001.png>
>>
>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Date: Tuesday, December 13, 2016 at 4:44 PM
>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Subject: Re: benchmark ATS
>>
>> Hi Steve,
>>
>> We had those things before I made the post, and so far I stuck with the same num.
>>
>> Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
>>
>> https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html <https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html>
>>
>>
>> Thanks,
>> Di Li
>>
>>
>>
>>
>> On Dec 13, 2016, at 11:08 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>
>> We use Ubuntu Server and in the end the only tuning was:
>>
>> · /etc/sysctl.conf
>> o net.ipv4.tcp_tw_recycle = 1
>> o net.core.somaxconn = 65535
>> o net.ipv4.tcp_fin_timeout = 15
>> o net.ipv4.tcp_keepalive_time = 300
>> o net.ipv4.tcp_keepalive_probes = 5
>> o net.ipv4.tcp_keepalive_intvl = 15
>>
>> But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
>>
>> The test was simply:
>>
>> ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache <http://[apache> traffic server forward proxy IP]:8080/index.html
>>
>> where post.txt is the file to post.
>>
>> You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
>>
>> To see the performance, we used commands ss-s and tops.
>> Run these on all the machine involved to keep an eye on everything.
>>
>> This was all run manually and quickly.
>>
>> -Steve
>>
>>
>>
>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>> <image001.png>
>>
>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Date: Tuesday, December 13, 2016 at 1:20 PM
>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Subject: Re: benchmark ATS
>>
>> Hey Steve,
>>
>> Can you share some details on config or performance turning or results ?
>>
>>
>> Thanks,
>> Di Li
>>
>>
>>
>>
>> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>>
>> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
>> I used Apache Bench for this.
>>
>> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
>> <image001.png>
>>
>> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
>> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Date: Tuesday, December 13, 2016 at 12:32 PM
>> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
>> Subject: Re: benchmark ATS
>>
>> using 6.2.0, repeatable
>>
>>
>> Thanks,
>> Di Li
>>
>>
>>
>>
>> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>>
>>
>> Am 13.12.2016 um 09:45 schrieb Di Li:
>>
>>
>>
>> When I doing some benchmark for outbound proxy, and has http_cache
>> enabled, well, first of all, the performance are pretty low, I guess I
>> didn’t do it right with the cache enabled, 2nd when I use wrk to have
>> 512 connection with 40 thread to go through proxy with http, it cause a
>> core dump, here’s the trace
>>
>> And when I disable the http.cache, the performance has went up a lot,
>> and no more coredump at all.
>>
>>
>> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
>> `w->alternate.valid()`
>> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
>> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
>> traffic_server - STACK TRACE
>>
>> is this repeatable?
>> which version of ATS?
>>
>> at least mention the software version should be common-sense
>>
>> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>>
>>
>>
>
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
Hi Steve,
Thanks for that info, I’m trying to use the small packets, each of request are just 200 byes, and responds in 900 bytes.
I don’t worry too much about the bandwidth, the request per second is my primary testing object, I only use 1 server as client, 1 server as proxy, 1 server as destination, all dedicated boxes
So far, I can’t pass 50000 req/s , I can see our proxy box has burned to 100% cpu usage with 24 cores. I’m still investigating if anything wrong on the config or the sysctl or IRQ balance
Thanks,
Di Li
> On Dec 13, 2016, at 2:01 PM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Are you using multiple VMs to post via a single VM with ATS to a tuned HTTPD?
>
> I got upwards of 50K connections and 9gbps. I posted giant binary files as well… my concern isn’t connections per second its bandwidth throttling which doesn’t seem to happen with these massive posts.
>
> In today’s age of VMs if we need more hits/sec we’d just spin up more VMs with ATS.
>
>
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
> Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Date: Tuesday, December 13, 2016 at 4:44 PM
> To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Subject: Re: benchmark ATS
>
> Hi Steve,
>
> We had those things before I made the post, and so far I stuck with the same num.
>
> Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
>
> https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html <https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html>
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 11:08 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> We use Ubuntu Server and in the end the only tuning was:
>
> · /etc/sysctl.conf
> o net.ipv4.tcp_tw_recycle = 1
> o net.core.somaxconn = 65535
> o net.ipv4.tcp_fin_timeout = 15
> o net.ipv4.tcp_keepalive_time = 300
> o net.ipv4.tcp_keepalive_probes = 5
> o net.ipv4.tcp_keepalive_intvl = 15
>
> But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
>
> The test was simply:
>
> ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache <http://[apache> traffic server forward proxy IP]:8080/index.html
>
> where post.txt is the file to post.
>
> You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
>
> To see the performance, we used commands ss-s and tops.
> Run these on all the machine involved to keep an eye on everything.
>
> This was all run manually and quickly.
>
> -Steve
>
>
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Date: Tuesday, December 13, 2016 at 1:20 PM
> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Subject: Re: benchmark ATS
>
> Hey Steve,
>
> Can you share some details on config or performance turning or results ?
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
> I used Apache Bench for this.
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Date: Tuesday, December 13, 2016 at 12:32 PM
> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Subject: Re: benchmark ATS
>
> using 6.2.0, repeatable
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>
>
> Am 13.12.2016 um 09:45 schrieb Di Li:
>
>
>
> When I doing some benchmark for outbound proxy, and has http_cache
> enabled, well, first of all, the performance are pretty low, I guess I
> didn’t do it right with the cache enabled, 2nd when I use wrk to have
> 512 connection with 40 thread to go through proxy with http, it cause a
> core dump, here’s the trace
>
> And when I disable the http.cache, the performance has went up a lot,
> and no more coredump at all.
>
>
> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
> `w->alternate.valid()`
> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
> traffic_server - STACK TRACE
>
> is this repeatable?
> which version of ATS?
>
> at least mention the software version should be common-sense
>
> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>
>
>
Re: benchmark ATS
Posted by "Lerner, Steve" <sl...@ebay.com>.
Are you using multiple VMs to post via a single VM with ATS to a tuned HTTPD?
I got upwards of 50K connections and 9gbps. I posted giant binary files as well… my concern isn’t connections per second its bandwidth throttling which doesn’t seem to happen with these massive posts.
In today’s age of VMs if we need more hits/sec we’d just spin up more VMs with ATS.
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
[cid:image001.png@01D25562.96D20D30]
From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Date: Tuesday, December 13, 2016 at 4:44 PM
To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Subject: Re: benchmark ATS
Hi Steve,
We had those things before I made the post, and so far I stuck with the same num.
Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html
Thanks,
Di Li
On Dec 13, 2016, at 11:08 AM, Lerner, Steve <sl...@ebay.com>> wrote:
We use Ubuntu Server and in the end the only tuning was:
• /etc/sysctl.conf
o net.ipv4.tcp_tw_recycle = 1
o net.core.somaxconn = 65535
o net.ipv4.tcp_fin_timeout = 15
o net.ipv4.tcp_keepalive_time = 300
o net.ipv4.tcp_keepalive_probes = 5
o net.ipv4.tcp_keepalive_intvl = 15
But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
The test was simply:
ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache traffic server forward proxy IP]:8080/index.html
where post.txt is the file to post.
You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
To see the performance, we used commands ss-s and tops.
Run these on all the machine involved to keep an eye on everything.
This was all run manually and quickly.
-Steve
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
<image001.png>
From: <di...@apple.com>> on behalf of Di Li <di...@apple.com>>
Reply-To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Date: Tuesday, December 13, 2016 at 1:20 PM
To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Subject: Re: benchmark ATS
Hey Steve,
Can you share some details on config or performance turning or results ?
Thanks,
Di Li
On Dec 13, 2016, at 9:46 AM, Lerner, Steve <sl...@ebay.com>> wrote:
I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
I used Apache Bench for this.
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
<image001.png>
From: <di...@apple.com>> on behalf of Di Li <di...@apple.com>>
Reply-To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Date: Tuesday, December 13, 2016 at 12:32 PM
To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Subject: Re: benchmark ATS
using 6.2.0, repeatable
Thanks,
Di Li
On Dec 13, 2016, at 1:28 AM, Reindl Harald <h....@thelounge.net>> wrote:
Am 13.12.2016 um 09:45 schrieb Di Li:
When I doing some benchmark for outbound proxy, and has http_cache
enabled, well, first of all, the performance are pretty low, I guess I
didn’t do it right with the cache enabled, 2nd when I use wrk to have
512 connection with 40 thread to go through proxy with http, it cause a
core dump, here’s the trace
And when I disable the http.cache, the performance has went up a lot,
and no more coredump at all.
FATAL: CacheRead.cc<http://cacheread.cc/> <http://cacheread.cc<http://cacheread.cc/>>:249: failed assert
`w->alternate.valid()`
traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
traffic_server: Aborted (Signal sent by tkill() 20136 1001)
traffic_server - STACK TRACE
is this repeatable?
which version of ATS?
at least mention the software version should be common-sense
had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
Hi Steve,
We had those things before I made the post, and so far I stuck with the same num.
Btw, there is a good article talk about the tw_recycle and tw_reuse, you may want to check it out, sometime tw_recycle is evil
https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html <https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html>
Thanks,
Di Li
> On Dec 13, 2016, at 11:08 AM, Lerner, Steve <sl...@ebay.com> wrote:
>
> We use Ubuntu Server and in the end the only tuning was:
>
> · /etc/sysctl.conf
> o net.ipv4.tcp_tw_recycle = 1
> o net.core.somaxconn = 65535
> o net.ipv4.tcp_fin_timeout = 15
> o net.ipv4.tcp_keepalive_time = 300
> o net.ipv4.tcp_keepalive_probes = 5
> o net.ipv4.tcp_keepalive_intvl = 15
>
> But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
>
> The test was simply:
>
> ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache <http://[apache> traffic server forward proxy IP]:8080/index.html
>
> where post.txt is the file to post.
>
> You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
>
> To see the performance, we used commands ss-s and tops.
> Run these on all the machine involved to keep an eye on everything.
>
> This was all run manually and quickly.
>
> -Steve
>
>
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
> Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Date: Tuesday, December 13, 2016 at 1:20 PM
> To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Subject: Re: benchmark ATS
>
> Hey Steve,
>
> Can you share some details on config or performance turning or results ?
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
> I used Apache Bench for this.
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di_li@apple.com <ma...@apple.com>> on behalf of Di Li <di_li@apple.com <ma...@apple.com>>
> Reply-To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Date: Tuesday, December 13, 2016 at 12:32 PM
> To: "users@trafficserver.apache.org <ma...@trafficserver.apache.org>" <users@trafficserver.apache.org <ma...@trafficserver.apache.org>>
> Subject: Re: benchmark ATS
>
> using 6.2.0, repeatable
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>
>
> Am 13.12.2016 um 09:45 schrieb Di Li:
>
>
> When I doing some benchmark for outbound proxy, and has http_cache
> enabled, well, first of all, the performance are pretty low, I guess I
> didn’t do it right with the cache enabled, 2nd when I use wrk to have
> 512 connection with 40 thread to go through proxy with http, it cause a
> core dump, here’s the trace
>
> And when I disable the http.cache, the performance has went up a lot,
> and no more coredump at all.
>
>
> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
> `w->alternate.valid()`
> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
> traffic_server - STACK TRACE
>
> is this repeatable?
> which version of ATS?
>
> at least mention the software version should be common-sense
>
> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>
>
Re: benchmark ATS
Posted by "Lerner, Steve" <sl...@ebay.com>.
We use Ubuntu Server and in the end the only tuning was:
· /etc/sysctl.conf
o net.ipv4.tcp_tw_recycle = 1
o net.core.somaxconn = 65535
o net.ipv4.tcp_fin_timeout = 15
o net.ipv4.tcp_keepalive_time = 300
o net.ipv4.tcp_keepalive_probes = 5
o net.ipv4.tcp_keepalive_intvl = 15
But of that batch I only think that SOMAXCONN made the difference. Try with just that tuning and then add the rest.
The test was simply:
ab -p post.txt -l -r -n 1000000 -c 20000 -k -H "Host: [apache httpd server IP]" http://[apache traffic server forward proxy IP]:8080/index.html
where post.txt is the file to post.
You can study apache bench manpage to understand the fields used and vary them to see the results. I’d use multiple client VMs running posts via apache bench targeting the single proxy server and be able to easily hit 9gbps and above.
To see the performance, we used commands ss-s and tops.
Run these on all the machine involved to keep an eye on everything.
This was all run manually and quickly.
-Steve
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
[cid:image001.png@01D2554A.6D35D0F0]
From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Date: Tuesday, December 13, 2016 at 1:20 PM
To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Subject: Re: benchmark ATS
Hey Steve,
Can you share some details on config or performance turning or results ?
Thanks,
Di Li
On Dec 13, 2016, at 9:46 AM, Lerner, Steve <sl...@ebay.com>> wrote:
I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
I used Apache Bench for this.
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
<image001.png>
From: <di...@apple.com>> on behalf of Di Li <di...@apple.com>>
Reply-To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Date: Tuesday, December 13, 2016 at 12:32 PM
To: "users@trafficserver.apache.org<ma...@trafficserver.apache.org>" <us...@trafficserver.apache.org>>
Subject: Re: benchmark ATS
using 6.2.0, repeatable
Thanks,
Di Li
On Dec 13, 2016, at 1:28 AM, Reindl Harald <h....@thelounge.net>> wrote:
Am 13.12.2016 um 09:45 schrieb Di Li:
When I doing some benchmark for outbound proxy, and has http_cache
enabled, well, first of all, the performance are pretty low, I guess I
didn’t do it right with the cache enabled, 2nd when I use wrk to have
512 connection with 40 thread to go through proxy with http, it cause a
core dump, here’s the trace
And when I disable the http.cache, the performance has went up a lot,
and no more coredump at all.
FATAL: CacheRead.cc<http://cacheread.cc/> <http://cacheread.cc<http://cacheread.cc/>>:249: failed assert
`w->alternate.valid()`
traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
traffic_server: Aborted (Signal sent by tkill() 20136 1001)
traffic_server - STACK TRACE
is this repeatable?
which version of ATS?
at least mention the software version should be common-sense
had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
Hey Steve,
Can you share some details on config or performance turning or results ?
Thanks,
Di Li
> On Dec 13, 2016, at 9:46 AM, Lerner, Steve <sl...@ebay.com> wrote:
>
> I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
> I used Apache Bench for this.
>
> Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com <ma...@ebay.com>
> <image001.png>
>
> From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
> Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Date: Tuesday, December 13, 2016 at 12:32 PM
> To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
> Subject: Re: benchmark ATS
>
> using 6.2.0, repeatable
>
>
> Thanks,
> Di Li
>
>
>
>
> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h.reindl@thelounge.net <ma...@thelounge.net>> wrote:
>
>
> Am 13.12.2016 um 09:45 schrieb Di Li:
>
> When I doing some benchmark for outbound proxy, and has http_cache
> enabled, well, first of all, the performance are pretty low, I guess I
> didn’t do it right with the cache enabled, 2nd when I use wrk to have
> 512 connection with 40 thread to go through proxy with http, it cause a
> core dump, here’s the trace
>
> And when I disable the http.cache, the performance has went up a lot,
> and no more coredump at all.
>
>
> FATAL: CacheRead.cc <http://cacheread.cc/> <http://cacheread.cc <http://cacheread.cc/>>:249: failed assert
> `w->alternate.valid()`
> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
> traffic_server - STACK TRACE
>
> is this repeatable?
> which version of ATS?
>
> at least mention the software version should be common-sense
>
> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
>
Re: benchmark ATS
Posted by "Lerner, Steve" <sl...@ebay.com>.
I’ve benchmarked ATS forward proxy post with cache disabled to near 10gbps on an Openstack VM with a 10Gbps NIC.
I used Apache Bench for this.
Steve Lerner | Director / Architect - Performance Engineering | m 212.495.9212 | slerner@ebay.com<ma...@ebay.com>
[cid:image001.png@01D2553E.DBABAC00]
From: <di...@apple.com> on behalf of Di Li <di...@apple.com>
Reply-To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Date: Tuesday, December 13, 2016 at 12:32 PM
To: "users@trafficserver.apache.org" <us...@trafficserver.apache.org>
Subject: Re: benchmark ATS
using 6.2.0, repeatable
Thanks,
Di Li
On Dec 13, 2016, at 1:28 AM, Reindl Harald <h....@thelounge.net>> wrote:
Am 13.12.2016 um 09:45 schrieb Di Li:
When I doing some benchmark for outbound proxy, and has http_cache
enabled, well, first of all, the performance are pretty low, I guess I
didn’t do it right with the cache enabled, 2nd when I use wrk to have
512 connection with 40 thread to go through proxy with http, it cause a
core dump, here’s the trace
And when I disable the http.cache, the performance has went up a lot,
and no more coredump at all.
FATAL: CacheRead.cc<http://cacheread.cc> <http://cacheread.cc>:249: failed assert
`w->alternate.valid()`
traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
traffic_server: Aborted (Signal sent by tkill() 20136 1001)
traffic_server - STACK TRACE
is this repeatable?
which version of ATS?
at least mention the software version should be common-sense
had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
Re: benchmark ATS
Posted by Di Li <di...@apple.com>.
using 6.2.0, repeatable
Thanks,
Di Li
> On Dec 13, 2016, at 1:28 AM, Reindl Harald <h....@thelounge.net> wrote:
>
>
> Am 13.12.2016 um 09:45 schrieb Di Li:
>> When I doing some benchmark for outbound proxy, and has http_cache
>> enabled, well, first of all, the performance are pretty low, I guess I
>> didn’t do it right with the cache enabled, 2nd when I use wrk to have
>> 512 connection with 40 thread to go through proxy with http, it cause a
>> core dump, here’s the trace
>>
>> And when I disable the http.cache, the performance has went up a lot,
>> and no more coredump at all.
>>
>>
>> FATAL: CacheRead.cc <http://cacheread.cc>:249: failed assert
>> `w->alternate.valid()`
>> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
>> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
>> traffic_server - STACK TRACE
>
> is this repeatable?
> which version of ATS?
>
> at least mention the software version should be common-sense
>
> had one such crash after upgrade to 7.0.0 and was not able to reproduce it, even not with a "ab -k -n 10000000 -c 500" benchmark
Re: benchmark ATS
Posted by Reindl Harald <h....@thelounge.net>.
Am 13.12.2016 um 09:45 schrieb Di Li:
> When I doing some benchmark for outbound proxy, and has http_cache
> enabled, well, first of all, the performance are pretty low, I guess I
> didn\u2019t do it right with the cache enabled, 2nd when I use wrk to have
> 512 connection with 40 thread to go through proxy with http, it cause a
> core dump, here\u2019s the trace
>
> And when I disable the http.cache, the performance has went up a lot,
> and no more coredump at all.
>
>
> FATAL: CacheRead.cc <http://cacheread.cc>:249: failed assert
> `w->alternate.valid()`
> traffic_server: using root directory '/ngs/app/oproxy/trafficserver'
> traffic_server: Aborted (Signal sent by tkill() 20136 1001)
> traffic_server - STACK TRACE
is this repeatable?
which version of ATS?
at least mention the software version should be common-sense
had one such crash after upgrade to 7.0.0 and was not able to reproduce
it, even not with a "ab -k -n 10000000 -c 500" benchmark