You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Knut Anders Hatlen (JIRA)" <ji...@apache.org> on 2011/04/27 13:39:03 UTC
[jira] [Created] (DERBY-5206) Investigate reduced throughput in
some of the nightly performance tests
Investigate reduced throughput in some of the nightly performance tests
-----------------------------------------------------------------------
Key: DERBY-5206
URL: https://issues.apache.org/jira/browse/DERBY-5206
Project: Derby
Issue Type: Task
Components: SQL
Affects Versions: 10.8.1.2
Environment: Solaris 10 5/08 s10x_u5wos_10 X86
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
Reporter: Knut Anders Hatlen
The last two months, the nightly performance regression tests (http://home.online.no/~olmsan/derby/perf/) have shown results that are slightly lower than normal for the following test:
- Single Record Select, Non-primary non-indexed column - http://home.online.no/~olmsan/derby/perf/select_sec_noindex_1y.html
- TPC-B like load (aka bank_tx) - http://home.online.no/~olmsan/derby/perf/tpcb_1y.html
For example, when averaging the results from 2011-03-01 to 2011-04-26 and comparing them to the average for the two preceding months (2011-01-01 to 2011-03-01), I see these changes:
Select, single-threaded: 17.05 tx/s -> 16.81 tx/s (down 1.4%)
Select, 4 threads: 25.50 tx/s -> 24.70 tx/s (down 3.1%)
Select, 16 threads: 25.17 tx/s -> 24.47 tx/s (down 2.8%)
bank_tx, single-threaded: 1931 tx/s -> 1904 tx/s (down 1.4%)
bank_tx, 4 threads: 2628 tx/s -> 2589 tx/s (down 1.5%)
bank_tx, 16 threads: 3584 tx/s -> 3519 tx/s (down 1.8%)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (DERBY-5206) Investigate reduced throughput in some
of the nightly performance tests
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Knut Anders Hatlen closed DERBY-5206.
-------------------------------------
Resolution: Invalid
I've had a very hard time reproducing this, but at last I remembered that the test itself was changed in DERBY-5067 around the time that it started happening. So now, if I run the test with Derby 10.8.1.2, I am able to see that the throughput is slightly higher if I take derbyTesting.jar from 10.7.1.1 (before DERBY-5067) than it is with derbyTesting.jar from 10.8.1.2.
I only see this if I run the test with the -init parameter to create a new test table in the same process that's running the actual test. If I run without -init, so that the test table from a previous run is reused, I don't see any difference in the throughput.
The changes in DERBY-5067 only affect how the test table is created (the primary key constraint is now added after the table has been populated, not when the table is created), which would explain why we only see it when running with -init. But the exact reason why is a bit of a mystery.
This issue sounds very similar to DERBY-4233. There we saw a drop in throughput in the table-scan test (same test as here) only when -init was specified, and the change that triggered the drop was a seemingly harmless change in a code path only executed when populating the test table.
The theory in that issue was that the runtime compiler in the JVM would optimize some code path based on statistics collected when populating the test table. The changes in how the table was populated would change these statistics slightly, possibly making the optimizer make different choices than before. Although these choices may be reasonable when populating the table, they may not be that good when the test starts doing selects instead of inserts, and it's the performance of the selects we're measuring in the test.
I think this is a theory that would explain what we're seeing in this issue too, and in any case, since the performance drop happened after a change in the test itself, there shouldn't be anything to fix in the product code. So I'll close the issue as invalid.
(After DERBY-4233 I was planning to change the nightly tests to create the test tables in a separate process to avoid this noise. Still on my todo list...)
> Investigate reduced throughput in some of the nightly performance tests
> -----------------------------------------------------------------------
>
> Key: DERBY-5206
> URL: https://issues.apache.org/jira/browse/DERBY-5206
> Project: Derby
> Issue Type: Task
> Components: SQL
> Affects Versions: 10.8.1.2
> Environment: Solaris 10 5/08 s10x_u5wos_10 X86
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build 1.6.0-b105)
> Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
> Reporter: Knut Anders Hatlen
>
> The last two months, the nightly performance regression tests (http://home.online.no/~olmsan/derby/perf/) have shown results that are slightly lower than normal for the following test:
> - Single Record Select, Non-primary non-indexed column - http://home.online.no/~olmsan/derby/perf/select_sec_noindex_1y.html
> - TPC-B like load (aka bank_tx) - http://home.online.no/~olmsan/derby/perf/tpcb_1y.html
> For example, when averaging the results from 2011-03-01 to 2011-04-26 and comparing them to the average for the two preceding months (2011-01-01 to 2011-03-01), I see these changes:
> Select, single-threaded: 17.05 tx/s -> 16.81 tx/s (down 1.4%)
> Select, 4 threads: 25.50 tx/s -> 24.70 tx/s (down 3.1%)
> Select, 16 threads: 25.17 tx/s -> 24.47 tx/s (down 2.8%)
> bank_tx, single-threaded: 1931 tx/s -> 1904 tx/s (down 1.4%)
> bank_tx, 4 threads: 2628 tx/s -> 2589 tx/s (down 1.5%)
> bank_tx, 16 threads: 3584 tx/s -> 3519 tx/s (down 1.8%)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DERBY-5206) Investigate reduced throughput in
some of the nightly performance tests
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-5206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025750#comment-13025750 ]
Knut Anders Hatlen commented on DERBY-5206:
-------------------------------------------
The tests are invoked using the following command line in the nightly testing (for the single-threaded variants):
java -Dderby.storage.pageCacheSize=25000 org.apache.derbyTesting.perf.clients.Runner -init -load sr_select -load_opts nonIndexed -wt 60 -rt 600
java -Dderby.storage.pageCacheSize=25000 org.apache.derbyTesting.perf.clients.Runner -init -load bank_tx -wt 60 -rt 600
> Investigate reduced throughput in some of the nightly performance tests
> -----------------------------------------------------------------------
>
> Key: DERBY-5206
> URL: https://issues.apache.org/jira/browse/DERBY-5206
> Project: Derby
> Issue Type: Task
> Components: SQL
> Affects Versions: 10.8.1.2
> Environment: Solaris 10 5/08 s10x_u5wos_10 X86
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build 1.6.0-b105)
> Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
> Reporter: Knut Anders Hatlen
>
> The last two months, the nightly performance regression tests (http://home.online.no/~olmsan/derby/perf/) have shown results that are slightly lower than normal for the following test:
> - Single Record Select, Non-primary non-indexed column - http://home.online.no/~olmsan/derby/perf/select_sec_noindex_1y.html
> - TPC-B like load (aka bank_tx) - http://home.online.no/~olmsan/derby/perf/tpcb_1y.html
> For example, when averaging the results from 2011-03-01 to 2011-04-26 and comparing them to the average for the two preceding months (2011-01-01 to 2011-03-01), I see these changes:
> Select, single-threaded: 17.05 tx/s -> 16.81 tx/s (down 1.4%)
> Select, 4 threads: 25.50 tx/s -> 24.70 tx/s (down 3.1%)
> Select, 16 threads: 25.17 tx/s -> 24.47 tx/s (down 2.8%)
> bank_tx, single-threaded: 1931 tx/s -> 1904 tx/s (down 1.4%)
> bank_tx, 4 threads: 2628 tx/s -> 2589 tx/s (down 1.5%)
> bank_tx, 16 threads: 3584 tx/s -> 3519 tx/s (down 1.8%)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira