You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Daniel John Debrunner <dj...@apache.org> on 2007/07/21 06:02:18 UTC

network server overhead

I ran a simple test of executing VALUES 1 using a prepared statement 
over the network server in auto-commit mode. 10.3 seemed to be the same 
performance as 10.2, maybe up to 4% slower. The 10.2 numbers for me were 
consistent, but the 10.3 numbers seemed to vary from 96% to 100% of the 
10.2 numbers.

I was actually surprised because I thought I seen some claims of much 
faster performance with 10.3 server.

I was getting around 1,630 transactions per second (each transaction is 
a VALUES 1 statement) on a 100Mbit network.

Did anyone make changes in 10.3 that they think would make this simple 
test improve, or is anyone working on reducing the basic overhead for 
client server? Are the nio changes intended to improve performance?

Thanks,
Dan.

Re: network server overhead

Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Daniel John Debrunner <dj...@apache.org> writes:

> I ran a simple test of executing VALUES 1 using a prepared statement
> over the network server in auto-commit mode. 10.3 seemed to be the
> same performance as 10.2, maybe up to 4% slower. The 10.2 numbers for
> me were consistent, but the 10.3 numbers seemed to vary from 96% to
> 100% of the 10.2 numbers.
>
> I was actually surprised because I thought I seen some claims of much
> faster performance with 10.3 server.
>
> I was getting around 1,630 transactions per second (each transaction
> is a VALUES 1 statement) on a 100Mbit network.

I tried to run a similar test with VALUES 1 in auto-commit mode over the
network (the actual test is attached as Values1.java). The test included
30 runs of each of the following configurations:

  - Derby 10.2.2.0 (no security manager)
  - Derby 10.3.1.4 (with security manager)
  - Derby 10.3.1.4 (without security manager)

All tests were run on Solaris 10 with Sun Java SE 6, both on the server
side and on the client side. Each run had a warm-up period of 45 seconds
and a steady-state period of 60 seconds. The network server was
restarted between each run.

What I noticed, was that

  a) the variability was about the same for 10.2.2.0 and 10.3.1.4

  b) any difference between 10.2 and 10.3 was small compared to the
     noise

  c) on average for all 30 runs, 10.3 with security manager had 0.64%
     higher throughput than 10.2, and 10.3 without security manager had
     0.68% higher throughput than 10.2

  d) when looking at the throughput graphs (attached as values1.png),
     the peaks for 10.3 seem to be at a higher level than the peaks for
     10.2. The 13 best runs with 10.3 are better than any of the 10.2
     runs, and the 5 worst runs with 10.2 are worse than any of the 10.3
     runs. (Since there were twice as many runs with 10.3, these numbers
     are of course a little skewed, but even if you remove any one of
     the 10.3 batches, you get the 6 or 7 best runs with 10.3 and the 5
     worst runs with 10.2.)

These numbers are as I would have expected. Since the test is
single-threaded, and the work that needs to be done by the embedded
driver to execute VALUES 1 is very small, the test is completely bounded
by the network latency. The ~0.7% improvement I saw might be caused by
reduced CPU usage in the 10.3 engine, but it could just as well be
noise. I would however expect a larger improvement if the VALUES 1 test
were run with a larger number of concurrent users, since then we might
get closer to saturating the CPU on the server.

It might be possible to improve single-threaded network server
performance for the VALUES 1 test by reducing the CPU usage, but I think
it would be hard to get a noticeable improvement without reducing the
number of round-trips. Currently, there are two round-trips (one to
fetch data and one to (auto-)commit). I don't think any of them can be
skipped easily, though.

-- 
Knut Anders

Re: network server overhead

Posted by Dy...@Sun.COM.
Daniel John Debrunner <dj...@apache.org> writes:

> I ran a simple test of executing VALUES 1 using a prepared statement
> over the network server in auto-commit mode. 10.3 seemed to be the
> same performance as 10.2, maybe up to 4% slower. The 10.2 numbers for
> me were consistent, but the 10.3 numbers seemed to vary from 96% to
> 100% of the 10.2 numbers.

I see a similar performance drop (see attached PDF file). (The
variablility seems to be about the same though).

> I was actually surprised because I thought I seen some claims of much
> faster performance with 10.3 server.
>
> I was getting around 1,630 transactions per second (each transaction
> is a VALUES 1 statement) on a 100Mbit network.
>
> Did anyone make changes in 10.3 that they think would make this simple
> test improve, or is anyone working on reducing the basic overhead for
> client server? 

> Are the nio changes intended to improve performance?

If you're referring to Knut's work I think the answer is yes, but
initially he could not see much improvement. (Knut is on vacation for
the next 3 weeks and may not read derby-dev mail that often).

-- 
dt


Re: network server overhead

Posted by Daniel John Debrunner <dj...@apache.org>.
Knut Anders Hatlen wrote:
> Daniel John Debrunner wrote:
>> I ran a simple test of executing VALUES 1 using a prepared statement 
>> over the network server in auto-commit mode. 10.3 seemed to be the 
>> same performance as 10.2, maybe up to 4% slower. The 10.2 numbers for 
>> me were consistent, but the 10.3 numbers seemed to vary from 96% to 
>> 100% of the 10.2 numbers.
> 
> Did you run the 10.3 tests with -noSecurityManager? It might be that the 
> security manager that is installed by default in 10.3 affects the 
> network server performance.

I had to because the network server didn't work with the security 
manager and remote hosts. :-)

The security manager should not cause performance degradation during 
steady state. This is because during steady state no methods should be 
called that require security checks.

Dan.


Re: network server overhead

Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Daniel John Debrunner wrote:
> I ran a simple test of executing VALUES 1 using a prepared statement 
> over the network server in auto-commit mode. 10.3 seemed to be the same 
> performance as 10.2, maybe up to 4% slower. The 10.2 numbers for me were 
> consistent, but the 10.3 numbers seemed to vary from 96% to 100% of the 
> 10.2 numbers.

Did you run the 10.3 tests with -noSecurityManager? It might be that the 
security manager that is installed by default in 10.3 affects the 
network server performance.

-- 
Knut Anders

Re: network server overhead

Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Daniel John Debrunner wrote:
> I ran a simple test of executing VALUES 1 using a prepared statement 
> over the network server in auto-commit mode. 10.3 seemed to be the same 
> performance as 10.2, maybe up to 4% slower. The 10.2 numbers for me were 
> consistent, but the 10.3 numbers seemed to vary from 96% to 100% of the 
> 10.2 numbers.
> 
> I was actually surprised because I thought I seen some claims of much 
> faster performance with 10.3 server.

I don't remember seeing any performance improvements specific to the 
network server code being checked in during the 10.3 development.

> I was getting around 1,630 transactions per second (each transaction is 
> a VALUES 1 statement) on a 100Mbit network.
> 
> Did anyone make changes in 10.3 that they think would make this simple 
> test improve,

Shouldn't the VALUES 1 test benefit from DERBY-827?

 > or is anyone working on reducing the basic overhead for
> client server? Are the nio changes intended to improve performance?

I did some profiling of the network server (trunk, not 10.3) two weeks 
ago to see if something jumped at me. I used a similar test with VALUES 
1 using prepared statement, as well as a test which returned a 
varchar(100) column. The motivation for the nio changes was that I 
noticed that temporary byte arrays were created each time DDMWriter 
wrote a string, whereas nio allowed encoding directly into the output 
buffer. For a query that returned a string of 100 ASCII characters, this 
change reduced the object allocations with ~600 bytes/query according to 
DTrace.

I also noticed that most of the network server's CPU went into 
SocketInputStream.read() (big surprise...), so I experimented with 
replacing the use of Socket and InputStream in DDMReader with 
SocketChannel and direct ByteBuffer (which is supposed to reduce 
internal copying between buffers), but I didn't see any effect of that 
change.

I don't have the profiling data here right now, but I can post them on 
the wiki when I get back from vacation in two weeks, to let others see 
if they can get something out of the numbers.

-- 
Knut Anders