You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Michele Sciabarra <mi...@sciabarra.com> on 2018/11/24 14:25:50 UTC
Performance Comparison of Runtimes and Improvements on the ActionLoop
Runtime
Hello all,
last week I discovered that, while the Python runtime is pretty slow and using an ActionLoop runtime improves Python a lot, the NodeJS runtime is damn fast, and the ActionLoop/GoLang runtime is way behind it.
So I tried to understand what is going on and why there is so much difference between ActionLoop and NodeJS
Most notably I decided to try to get rid of GoRoutines and just to launch a subprocess from an HTTP server and feed data to it.
Hence I wrote a quick-and-dirty prototype to see how fast I can go.
Those are the numbers, comparing raw performances of runtimes.
I measured performances in a similar environment. I run all the tests in Docker for Mac on a recent MacBook Pro,
using an "hello" action using the command 'wrk -t1 -c1 http://locahost:8080/run' that run a single thread single connection test for 10 seconds.
*** nodejs6action
Requests/sec: 1181.36
Transfer/sec: 264.19KB
*** python3action
Requests/sec: 19.80
Transfer/sec: 2.58KB
*** actionloop-golang-v1.11
Requests/sec: 83.80
Transfer/sec: 11.95KB
*** PROTO new actionloop-golang-v1.11
Requests/sec: 1031.38
Transfer/sec: 175.25KB
and this is the current
So while NodeJS will stay faster at 1181 req/sec (well, for a number of reasons), with ActionLoop (that can work with every language) we can get close at 1031 req/sec
Now I have to work in the runtime to implement the changes I made to the prototype
--
Michele Sciabarra
michele@sciabarra.com
Re: Performance Comparison of Runtimes and Improvements on the
ActionLoop Runtime
Posted by Michele Sciabarra <mi...@sciabarra.com>.
Updates on the performances of the GoLang runtime after a weekend of hacking .
The baseline is the Nodejs runtime:
wrk -t1 -c1 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.86ms 703.95us 18.73ms 98.62%
Req/Sec 1.22k 154.95 1.42k 82.00%
12177 requests in 10.00s, 2.65MB read
Requests/sec: 1217.30
Transfer/sec: 271.04KB
First, I cleaned the runtime removing Go Routines, and I reached those numbers:
wrk -t1 -c1 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 688.79us 173.93us 6.29ms 95.28%
Req/Sec 1.43k 217.80 1.68k 91.30%
6544 requests in 10.06s, 0.91MB read
Requests/sec: 650.42
Transfer/sec: 92.74KB
A bit better but still not entirely satisfactory. So I decided to try the plan A (the original plan!)
generate an action embedding the http server in the action. Luckily the infrastructure developed so far made it easy.
I also decided to follow Dragos suggestion using the fasthttp server. And this is the result!
wrk -t1 -c1 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
1 threads and 1 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 691.45us 1.28ms 37.10ms 99.42%
Req/Sec 1.61k 133.34 1.93k 80.20%
16132 requests in 10.10s, 2.66MB read
Requests/sec: 1597.22
Transfer/sec: 269.84KB
Yup, faster than nodejs. Also, this has the advantage that supports concurrency out of the box.
So I tried the performances with 10 concurrent requests, and compared with NodeJS with same level of concurrency:
NodeJS:
wrk -t10 -c10 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.78ms 1.71ms 27.45ms 77.25%
Req/Sec 269.66 83.11 555.00 70.20%
26875 requests in 10.01s, 5.84MB read
Requests/sec: 2684.94
Transfer/sec: 597.82KB
Golang Optimized:
wrk -t10 -c10 -stest.lua http://localhost:8080/run
Running 10s test @ http://localhost:8080/run
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.96ms 0.96ms 11.83ms 79.67%
Req/Sec 523.91 47.02 660.00 66.44%
52702 requests in 10.11s, 8.70MB read
Requests/sec: 5214.65
Transfer/sec: 0.86MB
5214 requests/second GoLang versus 2684.96 Node JS!
--
Michele Sciabarra
michele@sciabarra.com
----- Original message -----
From: Michele Sciabarra <mi...@sciabarra.com>
To: dev@openwhisk.apache.org
Subject: Performance Comparison of Runtimes and Improvements on the ActionLoop Runtime
Date: Sat, 24 Nov 2018 15:25:50 +0100
Hello all,
last week I discovered that, while the Python runtime is pretty slow and using an ActionLoop runtime improves Python a lot, the NodeJS runtime is damn fast, and the ActionLoop/GoLang runtime is way behind it.
So I tried to understand what is going on and why there is so much difference between ActionLoop and NodeJS
Most notably I decided to try to get rid of GoRoutines and just to launch a subprocess from an HTTP server and feed data to it.
Hence I wrote a quick-and-dirty prototype to see how fast I can go.
Those are the numbers, comparing raw performances of runtimes.
I measured performances in a similar environment. I run all the tests in Docker for Mac on a recent MacBook Pro,
using an "hello" action using the command 'wrk -t1 -c1 http://locahost:8080/run' that run a single thread single connection test for 10 seconds.
*** nodejs6action
Requests/sec: 1181.36
Transfer/sec: 264.19KB
*** python3action
Requests/sec: 19.80
Transfer/sec: 2.58KB
*** actionloop-golang-v1.11
Requests/sec: 83.80
Transfer/sec: 11.95KB
*** PROTO new actionloop-golang-v1.11
Requests/sec: 1031.38
Transfer/sec: 175.25KB
and this is the current
So while NodeJS will stay faster at 1181 req/sec (well, for a number of reasons), with ActionLoop (that can work with every language) we can get close at 1031 req/sec
Now I have to work in the runtime to implement the changes I made to the prototype
--
Michele Sciabarra
michele@sciabarra.com