You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "FAWC438 (via GitHub)" <gi...@apache.org> on 2023/03/03 06:34:38 UTC
[GitHub] [skywalking] FAWC438 commented on issue #10447: [Feature] Python agent performance enhancement with Confluent-Kafka
FAWC438 commented on issue #10447:
URL: https://github.com/apache/skywalking/issues/10447#issuecomment-1453048524
Now I have some bad news and good news.
The good news is that the performance improvement from confluent_kafka is substantial. Here are some of the results of my benchmark tests using [wrk](https://github.com/wg/wrk).
For the [fastapi-gunicorn use case](https://github.com/apache/skywalking-python/blob/master/demo/gunicorn_consumer_prefork.py) in the demo folder, I have done several benchmark tests in the same environment, and here are the more average results. You can see that confluent-kafka-python has **more than twice the performance improvement** compared to kafka-python.
_**kafka-python**_
```bash
[root@localhost wrk]# ./wrk -t36 -c4000 -d30s --latency http://10.3.242.223:8087/cat
Running 30s test @ http://10.3.242.223:8087/cat
36 threads and 4000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.41s 336.86ms 1.87s 72.58%
Req/Sec 44.25 68.59 570.00 89.08%
Latency Distribution
50% 1.58s
75% 1.62s
90% 1.86s
99% 1.86s
16667 requests in 30.05s, 2.65MB read
Socket errors: connect 0, read 5861, write 0, timeout 15241
Requests/sec: 554.71
Transfer/sec: 90.47KB
[root@localhost wrk]#
```
_**confluent-kafka-python**_
```bash
[root@localhost wrk]# ./wrk -t36 -c4000 -d30s --latency http://10.3.242.223:8086/cat
Running 30s test @ http://10.3.242.223:8086/cat
36 threads and 4000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.20s 426.34ms 1.98s 60.69%
Req/Sec 98.17 145.79 0.89k 87.40%
Latency Distribution
50% 1.16s
75% 1.56s
90% 1.79s
99% 1.92s
39687 requests in 30.05s, 6.32MB read
Socket errors: connect 0, read 3444, write 0, timeout 19562
Requests/sec: 1320.81
Transfer/sec: 215.41KB
[root@localhost wrk]#
```
Although kafka is **still a performance bottleneck**, such a high performance improvement is already a significant improvement for the program
```bash
Clock type: CPU
Ordered by: totaltime, desc
name ncall tsub ttot tavg
..py:321 SkyWalkingAgent.__heartbeat 2 0.000036 0.005567 0.002783
..afka.py:45 KafkaProtocol.heartbeat 2 0.000056 0.005531 0.002766
..ceManagementClient.send_heart_beat 2 0.000191 0.005465 0.002733
..ging/__init__.py:1424 Logger.debug 3 0.000058 0.005001 0.001667
..gging/__init__.py:1565 Logger._log 3 0.000089 0.004931 0.001644
..__init__.py:1550 Logger.makeRecord 3 0.000032 0.003934 0.001311
..__init__.py:282 LogRecord.__init__ 3 0.000249 0.003902 0.001301
..nt/kafka.py:132 heartbeat_callback 1 0.000006 0.003774 0.003774
..hon3.9/abc.py:96 __instancecheck__ 1 0.000017 0.003381 0.003381
..on3.9/abc.py:100 __subclasscheck__ 134/1 0.000170 0.003345 0.000025
...
Clock type: CPU
Ordered by: totaltime, desc
name ncall tsub ttot tavg
..335 SkyWalkingAgent.__report_meter 2 0.000032 0.001472 0.000736
...py:141 KafkaProtocol.report_meter 1 0.000069 0.001415 0.001415
..KafkaMeterDataReportService.report 1 0.000111 0.000730 0.000730
..ging/__init__.py:1424 Logger.debug 1 0.000035 0.000615 0.000615
..gging/__init__.py:1565 Logger._log 1 0.000052 0.000575 0.000575
..nt/protocol/kafka.py:144 generator 10 0.000067 0.000519 0.000052
..b/python3.9/queue.py:154 Queue.get 10 0.000105 0.000344 0.000034
..sw_logging.py:40 Logger._sw_handle 1 0.000010 0.000286 0.000286
..ing/__init__.py:1591 Logger.handle 1 0.000012 0.000277 0.000277
...9/threading.py:280 Condition.wait 3 0.000070 0.000275 0.000092
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org