You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by ma...@apache.org on 2024/01/08 17:33:01 UTC
(tomcat) branch main updated (72430c2072 -> d24302cb12)
This is an automated email from the ASF dual-hosted git repository.
markt pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
from 72430c2072 Add license
new 0b72ad2b00 Refactor absolute performance tests so they do not run by default
new 7a03eb4970 Add additional comment to existing absolute performance tests
new 32a57904e4 Refactor so these relative performance tests generate pass/fail results
new a574b5af78 Add additional comment to this relative performance test
new 6be9e259d6 Comment this absolute performance test with an upper limit
new 7a2352969b This test is very timing sensitive. Add some test specific margins.
new 8e05a017af Update the docs for running the tests
new d24302cb12 Don't test lazy vs full as they can be quite close
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
BUILDING.txt | 17 ++++++--
...a => TesterCompositeELResolverPerformance.java} | 6 ++-
.../jakarta/el/TesterImportHandlerPerformance.java | 4 ++
...ance.java => TesterHttpServletPerformance.java} | 6 +--
...esterScopedAttributeELResolverPerformance.java} | 6 ++-
.../TesterContainerProviderPerformance.java | 4 ++
.../TesterDigestAuthenticatorPerformance.java | 4 ++
.../connector/TestResponsePerformance.java | 8 +++-
...formance.java => TesterRequestPerformance.java} | 6 ++-
...> TesterApplicationHttpRequestPerformance.java} | 6 ++-
...> TesterDefaultInstanceManagerPerformance.java} | 9 +++--
.../catalina/mapper/TestMapperPerformance.java | 3 ++
... TesterAbstractFileResourceSetPerformance.java} | 6 ++-
.../apache/el/parser/TestELParserPerformance.java | 10 +++++
...e.java => TesterTagHandlerPoolPerformance.java} | 7 +++-
.../juli/TestOneLineFormatterPerformance.java | 31 +++++++++-----
...> TesterOneLineFormatterMillisPerformance.java} | 6 ++-
.../tomcat/unittest/TesterThreadedPerformance.java | 3 ++
.../apache/tomcat/util/bcel/TesterPerformance.java | 6 +++
.../util/buf/TestCharsetCachePerformance.java | 31 +++++++-------
.../tomcat/util/http/TesterCookiesPerformance.java | 4 ++
...va => TesterFastHttpDateFormatPerformance.java} | 6 ++-
.../util/http/TesterParametersPerformance.java | 4 ++
....java => TesterConnectionLimitPerformance.java} | 6 ++-
.../server/TestAsyncMessagesPerformance.java | 47 +++++++++++++++++-----
25 files changed, 190 insertions(+), 56 deletions(-)
rename test/jakarta/el/{TestCompositeELResolverPerformance.java => TesterCompositeELResolverPerformance.java} (88%)
rename test/jakarta/servlet/http/{TestHttpServletPerformance.java => TesterHttpServletPerformance.java} (90%)
rename test/jakarta/servlet/jsp/el/{TestScopedAttributeELResolverPerformance.java => TesterScopedAttributeELResolverPerformance.java} (87%)
rename test/org/apache/catalina/connector/{TestRequestPerformance.java => TesterRequestPerformance.java} (87%)
rename test/org/apache/catalina/core/{TestApplicationHttpRequestPerformance.java => TesterApplicationHttpRequestPerformance.java} (88%)
rename test/org/apache/catalina/core/{TestDefaultInstanceManagerPerformance.java => TesterDefaultInstanceManagerPerformance.java} (92%)
rename test/org/apache/catalina/webresources/{TestAbstractFileResourceSetPerformance.java => TesterAbstractFileResourceSetPerformance.java} (93%)
rename test/org/apache/jasper/runtime/{TestTagHandlerPoolPerformance.java => TesterTagHandlerPoolPerformance.java} (92%)
rename test/org/apache/juli/{TestOneLineFormatterMillisPerformance.java => TesterOneLineFormatterMillisPerformance.java} (92%)
rename test/org/apache/tomcat/util/http/{TestFastHttpDateFormatPerformance.java => TesterFastHttpDateFormatPerformance.java} (89%)
rename test/org/apache/tomcat/websocket/{TestConnectionLimitPerformance.java => TesterConnectionLimitPerformance.java} (93%)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 01/08: Refactor absolute performance tests so they do not run by default
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 0b72ad2b00207195441fb4bb9686c940eda32945
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:06:19 2024 +0000
Refactor absolute performance tests so they do not run by default
Absolute performance tests report the time taken to complete a series of
actions. They cannot fail therefore do not run them as part of a
standard test run.
These tests are excluded by naming them Tester...
---
...erformance.java => TesterCompositeELResolverPerformance.java} | 6 +++++-
...ServletPerformance.java => TesterHttpServletPerformance.java} | 6 +++---
...ance.java => TesterScopedAttributeELResolverPerformance.java} | 6 +++++-
...TestRequestPerformance.java => TesterRequestPerformance.java} | 6 +++++-
...ormance.java => TesterApplicationHttpRequestPerformance.java} | 6 +++++-
...ormance.java => TesterDefaultInstanceManagerPerformance.java} | 9 +++++----
...rmance.java => TesterAbstractFileResourceSetPerformance.java} | 6 +++++-
...PoolPerformance.java => TesterTagHandlerPoolPerformance.java} | 7 +++++--
...ormance.java => TesterOneLineFormatterMillisPerformance.java} | 6 +++++-
...Performance.java => TesterFastHttpDateFormatPerformance.java} | 6 +++++-
...mitPerformance.java => TesterConnectionLimitPerformance.java} | 6 +++++-
11 files changed, 53 insertions(+), 17 deletions(-)
diff --git a/test/jakarta/el/TestCompositeELResolverPerformance.java b/test/jakarta/el/TesterCompositeELResolverPerformance.java
similarity index 88%
rename from test/jakarta/el/TestCompositeELResolverPerformance.java
rename to test/jakarta/el/TesterCompositeELResolverPerformance.java
index f7974dd481..e974b979dd 100644
--- a/test/jakarta/el/TestCompositeELResolverPerformance.java
+++ b/test/jakarta/el/TesterCompositeELResolverPerformance.java
@@ -18,7 +18,11 @@ package jakarta.el;
import org.junit.Test;
-public class TestCompositeELResolverPerformance {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterCompositeELResolverPerformance {
/*
* https://bz.apache.org/bugzilla/show_bug.cgi?id=68119
diff --git a/test/jakarta/servlet/http/TestHttpServletPerformance.java b/test/jakarta/servlet/http/TesterHttpServletPerformance.java
similarity index 90%
rename from test/jakarta/servlet/http/TestHttpServletPerformance.java
rename to test/jakarta/servlet/http/TesterHttpServletPerformance.java
index 98526de898..aeb5cc4ca4 100644
--- a/test/jakarta/servlet/http/TestHttpServletPerformance.java
+++ b/test/jakarta/servlet/http/TesterHttpServletPerformance.java
@@ -27,10 +27,10 @@ import org.apache.catalina.filters.TesterHttpServletResponse;
/*
- * Note: Class name chosen so it matches *Performance.java and can be excluded
- * from test runs along with other performance tests.
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
*/
-public class TestHttpServletPerformance {
+public class TesterHttpServletPerformance {
@Test
public void testDoOptions() throws IOException, ServletException{
diff --git a/test/jakarta/servlet/jsp/el/TestScopedAttributeELResolverPerformance.java b/test/jakarta/servlet/jsp/el/TesterScopedAttributeELResolverPerformance.java
similarity index 87%
rename from test/jakarta/servlet/jsp/el/TestScopedAttributeELResolverPerformance.java
rename to test/jakarta/servlet/jsp/el/TesterScopedAttributeELResolverPerformance.java
index 341d44d45c..3b37ef1ca2 100644
--- a/test/jakarta/servlet/jsp/el/TestScopedAttributeELResolverPerformance.java
+++ b/test/jakarta/servlet/jsp/el/TesterScopedAttributeELResolverPerformance.java
@@ -25,7 +25,11 @@ import jakarta.servlet.jsp.TesterPageContext;
import org.junit.Test;
-public class TestScopedAttributeELResolverPerformance {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterScopedAttributeELResolverPerformance {
/*
* With the caching of NotFound responses this test takes ~20ms. Without the
diff --git a/test/org/apache/catalina/connector/TestRequestPerformance.java b/test/org/apache/catalina/connector/TesterRequestPerformance.java
similarity index 87%
rename from test/org/apache/catalina/connector/TestRequestPerformance.java
rename to test/org/apache/catalina/connector/TesterRequestPerformance.java
index 81a00829ce..42ec455051 100644
--- a/test/org/apache/catalina/connector/TestRequestPerformance.java
+++ b/test/org/apache/catalina/connector/TesterRequestPerformance.java
@@ -20,7 +20,11 @@ import org.junit.Test;
import org.apache.tomcat.unittest.TesterRequest;
-public class TestRequestPerformance {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterRequestPerformance {
@Test
public void localeParsePerformance() throws Exception {
diff --git a/test/org/apache/catalina/core/TestApplicationHttpRequestPerformance.java b/test/org/apache/catalina/core/TesterApplicationHttpRequestPerformance.java
similarity index 88%
rename from test/org/apache/catalina/core/TestApplicationHttpRequestPerformance.java
rename to test/org/apache/catalina/core/TesterApplicationHttpRequestPerformance.java
index a0a1556a8e..e25cf62479 100644
--- a/test/org/apache/catalina/core/TestApplicationHttpRequestPerformance.java
+++ b/test/org/apache/catalina/core/TesterApplicationHttpRequestPerformance.java
@@ -20,7 +20,11 @@ import org.junit.Test;
import org.apache.catalina.connector.Request;
-public class TestApplicationHttpRequestPerformance {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterApplicationHttpRequestPerformance {
@Test
public void testGetAttribute() {
diff --git a/test/org/apache/catalina/core/TestDefaultInstanceManagerPerformance.java b/test/org/apache/catalina/core/TesterDefaultInstanceManagerPerformance.java
similarity index 92%
rename from test/org/apache/catalina/core/TestDefaultInstanceManagerPerformance.java
rename to test/org/apache/catalina/core/TesterDefaultInstanceManagerPerformance.java
index c496efeb08..cbea78df9c 100644
--- a/test/org/apache/catalina/core/TestDefaultInstanceManagerPerformance.java
+++ b/test/org/apache/catalina/core/TesterDefaultInstanceManagerPerformance.java
@@ -28,11 +28,12 @@ import org.apache.catalina.startup.Tomcat;
import org.apache.catalina.startup.TomcatBaseTest;
import org.apache.tomcat.InstanceManager;
-public class TestDefaultInstanceManagerPerformance extends TomcatBaseTest {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterDefaultInstanceManagerPerformance extends TomcatBaseTest {
- /*
- * Performance test. Comment out @Ignore to run the test.
- */
@Test
public void testConcurrency() throws Exception {
// Create a populated InstanceManager
diff --git a/test/org/apache/catalina/webresources/TestAbstractFileResourceSetPerformance.java b/test/org/apache/catalina/webresources/TesterAbstractFileResourceSetPerformance.java
similarity index 93%
rename from test/org/apache/catalina/webresources/TestAbstractFileResourceSetPerformance.java
rename to test/org/apache/catalina/webresources/TesterAbstractFileResourceSetPerformance.java
index 5b17d116a1..7d66c129be 100644
--- a/test/org/apache/catalina/webresources/TestAbstractFileResourceSetPerformance.java
+++ b/test/org/apache/catalina/webresources/TesterAbstractFileResourceSetPerformance.java
@@ -20,7 +20,11 @@ import java.util.regex.Pattern;
import org.junit.Test;
-public class TestAbstractFileResourceSetPerformance {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterAbstractFileResourceSetPerformance {
private static final Pattern UNSAFE_WINDOWS_FILENAME_PATTERN = Pattern.compile(" $|[\"<>]");
diff --git a/test/org/apache/jasper/runtime/TestTagHandlerPoolPerformance.java b/test/org/apache/jasper/runtime/TesterTagHandlerPoolPerformance.java
similarity index 92%
rename from test/org/apache/jasper/runtime/TestTagHandlerPoolPerformance.java
rename to test/org/apache/jasper/runtime/TesterTagHandlerPoolPerformance.java
index 66515872e8..5c3b556477 100644
--- a/test/org/apache/jasper/runtime/TestTagHandlerPoolPerformance.java
+++ b/test/org/apache/jasper/runtime/TesterTagHandlerPoolPerformance.java
@@ -30,8 +30,11 @@ import org.apache.catalina.startup.TomcatBaseTest;
import org.apache.tomcat.unittest.TesterThreadedPerformance;
import org.apache.tomcat.unittest.tags.Bug53545;
-
-public class TestTagHandlerPoolPerformance extends TomcatBaseTest {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterTagHandlerPoolPerformance extends TomcatBaseTest {
@Test
public void testConcurrency() throws Exception {
diff --git a/test/org/apache/juli/TestOneLineFormatterMillisPerformance.java b/test/org/apache/juli/TesterOneLineFormatterMillisPerformance.java
similarity index 92%
rename from test/org/apache/juli/TestOneLineFormatterMillisPerformance.java
rename to test/org/apache/juli/TesterOneLineFormatterMillisPerformance.java
index b7c0cabd28..41dc3a8927 100644
--- a/test/org/apache/juli/TestOneLineFormatterMillisPerformance.java
+++ b/test/org/apache/juli/TesterOneLineFormatterMillisPerformance.java
@@ -25,8 +25,12 @@ import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
@RunWith(Parameterized.class)
-public class TestOneLineFormatterMillisPerformance {
+public class TesterOneLineFormatterMillisPerformance {
@Parameterized.Parameters(name = "{index}: format[{0}]")
public static Collection<Object[]> parameters() {
diff --git a/test/org/apache/tomcat/util/http/TestFastHttpDateFormatPerformance.java b/test/org/apache/tomcat/util/http/TesterFastHttpDateFormatPerformance.java
similarity index 89%
rename from test/org/apache/tomcat/util/http/TestFastHttpDateFormatPerformance.java
rename to test/org/apache/tomcat/util/http/TesterFastHttpDateFormatPerformance.java
index ad3e176cfb..4d6e579e91 100644
--- a/test/org/apache/tomcat/util/http/TestFastHttpDateFormatPerformance.java
+++ b/test/org/apache/tomcat/util/http/TesterFastHttpDateFormatPerformance.java
@@ -18,7 +18,11 @@ package org.apache.tomcat.util.http;
import org.junit.Test;
-public class TestFastHttpDateFormatPerformance {
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
+public class TesterFastHttpDateFormatPerformance {
@Test
public void testGetCurrentDateConcurrent() throws InterruptedException {
diff --git a/test/org/apache/tomcat/websocket/TestConnectionLimitPerformance.java b/test/org/apache/tomcat/websocket/TesterConnectionLimitPerformance.java
similarity index 93%
rename from test/org/apache/tomcat/websocket/TestConnectionLimitPerformance.java
rename to test/org/apache/tomcat/websocket/TesterConnectionLimitPerformance.java
index 6fc045622c..d5deb7ebf5 100644
--- a/test/org/apache/tomcat/websocket/TestConnectionLimitPerformance.java
+++ b/test/org/apache/tomcat/websocket/TesterConnectionLimitPerformance.java
@@ -33,7 +33,11 @@ import org.apache.catalina.startup.Tomcat;
import org.apache.catalina.startup.TomcatBaseTest;
import org.apache.tomcat.websocket.TesterMessageCountClient.TesterProgrammaticEndpoint;
-public class TestConnectionLimitPerformance extends TomcatBaseTest {
+/*
+ * This test runs until the test machine runs out of resources. There is no benefit it running it as part of a standard
+ * test run so it is excluded due to the name starting Tester...
+ */
+public class TesterConnectionLimitPerformance extends TomcatBaseTest {
/*
* Simple test to see how many outgoing connections can be created on a single machine.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 06/08: This test is very timing sensitive. Add some test specific margins.
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 7a2352969b484212fa230c5df97c9bd64e24d886
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:11:31 2024 +0000
This test is very timing sensitive. Add some test specific margins.
The current margins are based on local testing and reports on the dev
list. They may need to be relaxed further or if the results are very
inconsistent between systems it may need to be disabled by default.
---
.../server/TestAsyncMessagesPerformance.java | 47 +++++++++++++++++-----
1 file changed, 38 insertions(+), 9 deletions(-)
diff --git a/test/org/apache/tomcat/websocket/server/TestAsyncMessagesPerformance.java b/test/org/apache/tomcat/websocket/server/TestAsyncMessagesPerformance.java
index c169f61a60..5304bee2ca 100644
--- a/test/org/apache/tomcat/websocket/server/TestAsyncMessagesPerformance.java
+++ b/test/org/apache/tomcat/websocket/server/TestAsyncMessagesPerformance.java
@@ -36,6 +36,11 @@ import org.apache.catalina.startup.TomcatBaseTest;
import org.apache.tomcat.websocket.TesterAsyncTiming;
import org.apache.tomcat.websocket.TesterMessageCountClient.TesterProgrammaticEndpoint;
+/*
+ * This test is very timing sensitive. Any failures need to be checked to see if the thresholds just need adjusting to
+ * support a wider range of platforms and/or Java versions or if the failure is an indication of a performance drop in
+ * the WebSocket implementation.
+ */
public class TestAsyncMessagesPerformance extends TomcatBaseTest {
@Test
@@ -69,11 +74,20 @@ public class TestAsyncMessagesPerformance extends TomcatBaseTest {
private long lastMessage = 0;
private int sequence = 0;
private int count = 0;
+ private long seqZeroTimingFailureCount = 0;
+ private long seqOneTimingFailureCount = 0;
+ private long seqTwoTimingFailureCount = 0;
+
private CountDownLatch latch = new CountDownLatch(1);
private volatile boolean fail = false;
@Override
public void onMessage(ByteBuffer message, boolean last) {
+ // Expected received data is:
+ // 1 * 16k message in 2 * 8k chunks
+ // 1 * 4k message in 1 * 4k chunk
+ // 50 ms pause
+ // loop
if (lastMessage == 0) {
// First message. Don't check
sequence++;
@@ -87,34 +101,36 @@ public class TestAsyncMessagesPerformance extends TomcatBaseTest {
sequence++;
if (message.capacity() != 8192) {
System.out.println(
- "Expected size 8192 but was [" + message.capacity() + "], count [" + count + "]");
+ "SEQ0: Expected size 8192 but was [" + message.capacity() + "], count [" + count + "]");
fail = true;
}
if (diff < 40000000) {
- System.out.println("Expected diff > 40ms but was [" + diff + "], count [" + count + "]");
- fail = true;
+ System.out.println("SEQ0: Expected diff > 40ms but was [" + diff + "], count [" + count + "]");
+ seqZeroTimingFailureCount++;
}
} else if (sequence == 1) {
sequence++;
if (message.capacity() != 8192) {
System.out.println(
- "Expected size 8192 but was [" + message.capacity() + "], count [" + count + "]");
+ "SEQ1: Expected size 8192 but was [" + message.capacity() + "], count [" + count + "]");
fail = true;
}
+ // Gap between 2* 8k chunks of 16k message expected to be less than 0.5ms
if (diff > 500000) {
- System.out.println("Expected diff < 500,000 but was [" + diff + "], count [" + count + "]");
- fail = true;
+ System.out.println("SEQ1: Expected diff < 500,000 but was [" + diff + "], count [" + count + "]");
+ seqOneTimingFailureCount++;
}
} else if (sequence == 2) {
sequence = 0;
if (message.capacity() != 4096) {
System.out.println(
- "Expected size 4096 but was [" + message.capacity() + "], count [" + count + "]");
+ "SEQ2: Expected size 4096 but was [" + message.capacity() + "], count [" + count + "]");
fail = true;
}
+ // Gap between 16k message and 4k message expected to be less than 0.5ms
if (diff > 500000) {
- System.out.println("Expected diff < 500,000 but was [" + diff + "], count [" + count + "]");
- fail = true;
+ System.out.println("SEQ2: Expected diff < 500,000 but was [" + diff + "], count [" + count + "]");
+ seqTwoTimingFailureCount++;
}
}
}
@@ -130,6 +146,19 @@ public class TestAsyncMessagesPerformance extends TomcatBaseTest {
}
public boolean hasFailed() {
+ // Total iterations are 1500
+ if (!fail) {
+ if (seqZeroTimingFailureCount > 1) {
+ // The 50ms pause after the short message may very rarely appear to be less than 40ms
+ fail = true;
+ } else if (seqOneTimingFailureCount > 5) {
+ // The two chunks of the 16k message may rarely be more than 0.5ms apart
+ fail = true;
+ } else if (seqTwoTimingFailureCount > 100) {
+ // The short message may often be more than 0.5ms after the long message
+ fail = true;
+ }
+ }
return fail;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 07/08: Update the docs for running the tests
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 8e05a017af74642068b56cadec25f93ebbf75dad
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:11:41 2024 +0000
Update the docs for running the tests
---
BUILDING.txt | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/BUILDING.txt b/BUILDING.txt
index 0acef652a1..d4e248b7b0 100644
--- a/BUILDING.txt
+++ b/BUILDING.txt
@@ -492,9 +492,20 @@ NOTE: Cobertura is licensed under GPL v2 with parts of it being under
(such as a developer may use day to day) assuming no other resource hungry
processes are running.
- These assumptions are not always true (e.g. on CI systems running in a
- virtual machine) so the performance tests may be deactivated by using the
- following property:
+ Performance tests may be an absolute test (how long to complete a number
+ of iterations of an operation or set of operations) or may be a relative
+ test (how long two or more different approaches take to generate the same
+ result). The absolute tests may be destructive in that they run until the
+ system runs out of resources.
+
+ Where there is no benefit in running an absolute performance test as part
+ of a standard test run, the test will be excluded by naming it
+ Tester*Performance.java.
+
+ The relative tests are included as part of a standard test run however,
+ where the assumptions made about host capabilities are not true (e.g. on
+ CI systems running in virtual machine) the tests may be deactivated by
+ using the following property:
test.excludePerformance=true
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 02/08: Add additional comment to existing absolute performance tests
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 7a03eb497045752bd6d81b4326669d57affb2bdd
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:06:53 2024 +0000
Add additional comment to existing absolute performance tests
---
test/jakarta/el/TesterImportHandlerPerformance.java | 4 ++++
test/jakarta/websocket/TesterContainerProviderPerformance.java | 4 ++++
.../authenticator/TesterDigestAuthenticatorPerformance.java | 4 ++++
test/org/apache/tomcat/unittest/TesterThreadedPerformance.java | 3 +++
test/org/apache/tomcat/util/bcel/TesterPerformance.java | 6 ++++++
test/org/apache/tomcat/util/http/TesterCookiesPerformance.java | 4 ++++
test/org/apache/tomcat/util/http/TesterParametersPerformance.java | 4 ++++
7 files changed, 29 insertions(+)
diff --git a/test/jakarta/el/TesterImportHandlerPerformance.java b/test/jakarta/el/TesterImportHandlerPerformance.java
index c2c267c851..77e39e7624 100644
--- a/test/jakarta/el/TesterImportHandlerPerformance.java
+++ b/test/jakarta/el/TesterImportHandlerPerformance.java
@@ -18,6 +18,10 @@ package jakarta.el;
import org.junit.Test;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
public class TesterImportHandlerPerformance {
/*
diff --git a/test/jakarta/websocket/TesterContainerProviderPerformance.java b/test/jakarta/websocket/TesterContainerProviderPerformance.java
index a3c8a87f46..0e2c7181b3 100644
--- a/test/jakarta/websocket/TesterContainerProviderPerformance.java
+++ b/test/jakarta/websocket/TesterContainerProviderPerformance.java
@@ -23,6 +23,10 @@ import org.junit.Test;
import org.apache.tomcat.unittest.TesterThreadedPerformance;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
public class TesterContainerProviderPerformance {
@Test
diff --git a/test/org/apache/catalina/authenticator/TesterDigestAuthenticatorPerformance.java b/test/org/apache/catalina/authenticator/TesterDigestAuthenticatorPerformance.java
index ad417f2e5c..11bf9cd560 100644
--- a/test/org/apache/catalina/authenticator/TesterDigestAuthenticatorPerformance.java
+++ b/test/org/apache/catalina/authenticator/TesterDigestAuthenticatorPerformance.java
@@ -41,6 +41,10 @@ import org.apache.tomcat.util.buf.HexUtils;
import org.apache.tomcat.util.descriptor.web.LoginConfig;
import org.apache.tomcat.util.security.ConcurrentMessageDigest;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
public class TesterDigestAuthenticatorPerformance {
private static String USER = "user";
diff --git a/test/org/apache/tomcat/unittest/TesterThreadedPerformance.java b/test/org/apache/tomcat/unittest/TesterThreadedPerformance.java
index 2c3112ca69..9e0ea4c29e 100644
--- a/test/org/apache/tomcat/unittest/TesterThreadedPerformance.java
+++ b/test/org/apache/tomcat/unittest/TesterThreadedPerformance.java
@@ -19,6 +19,9 @@ package org.apache.tomcat.unittest;
import java.util.function.IntConsumer;
import java.util.function.Supplier;
+/*
+ * Support class for threaded performance tests.
+ */
public class TesterThreadedPerformance {
private final int threadCount;
diff --git a/test/org/apache/tomcat/util/bcel/TesterPerformance.java b/test/org/apache/tomcat/util/bcel/TesterPerformance.java
index b3400dc7d8..945637192c 100644
--- a/test/org/apache/tomcat/util/bcel/TesterPerformance.java
+++ b/test/org/apache/tomcat/util/bcel/TesterPerformance.java
@@ -32,6 +32,12 @@ import org.apache.tomcat.Jar;
import org.apache.tomcat.util.bcel.classfile.ClassParser;
import org.apache.tomcat.util.scan.JarFactory;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ *
+ * The test also requires that the Jira lib directory has been extracted from a Jira install and copied to /tmp.
+ */
public class TesterPerformance {
private static final String JAR_LOCATION = "/tmp/jira-libs";
diff --git a/test/org/apache/tomcat/util/http/TesterCookiesPerformance.java b/test/org/apache/tomcat/util/http/TesterCookiesPerformance.java
index 76c5d398fa..5504bde4ae 100644
--- a/test/org/apache/tomcat/util/http/TesterCookiesPerformance.java
+++ b/test/org/apache/tomcat/util/http/TesterCookiesPerformance.java
@@ -21,6 +21,10 @@ import org.junit.Test;
import org.apache.tomcat.util.buf.MessageBytes;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
public class TesterCookiesPerformance {
@Test
diff --git a/test/org/apache/tomcat/util/http/TesterParametersPerformance.java b/test/org/apache/tomcat/util/http/TesterParametersPerformance.java
index 17a7d5f8ec..abcc47f9e9 100644
--- a/test/org/apache/tomcat/util/http/TesterParametersPerformance.java
+++ b/test/org/apache/tomcat/util/http/TesterParametersPerformance.java
@@ -27,6 +27,10 @@ import org.junit.Test;
import org.apache.tomcat.util.buf.B2CConverter;
+/*
+ * This is an absolute performance test. There is no benefit it running it as part of a standard test run so it is
+ * excluded due to the name starting Tester...
+ */
public class TesterParametersPerformance {
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 03/08: Refactor so these relative performance tests generate pass/fail results
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 32a57904e46ce3f02efc532c4293e3487f3c5f7c
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:08:21 2024 +0000
Refactor so these relative performance tests generate pass/fail results
---
.../apache/el/parser/TestELParserPerformance.java | 10 +++++++
.../juli/TestOneLineFormatterPerformance.java | 31 +++++++++++++++-------
.../util/buf/TestCharsetCachePerformance.java | 30 ++++++++++-----------
3 files changed, 46 insertions(+), 25 deletions(-)
diff --git a/test/org/apache/el/parser/TestELParserPerformance.java b/test/org/apache/el/parser/TestELParserPerformance.java
index de2aa9d1e0..eb860f4be9 100644
--- a/test/org/apache/el/parser/TestELParserPerformance.java
+++ b/test/org/apache/el/parser/TestELParserPerformance.java
@@ -20,10 +20,14 @@ import java.io.StringReader;
import jakarta.el.ELBaseTest;
+import org.junit.Assert;
import org.junit.Test;
import org.apache.tomcat.util.collections.SynchronizedStack;
+/*
+ * This is a relative performance test so it remains part of the standard test run.
+ */
public class TestELParserPerformance extends ELBaseTest {
/*
@@ -48,6 +52,8 @@ public class TestELParserPerformance extends ELBaseTest {
final int runs = 20;
final int parseIterations = 10000;
+ long reinitTotalTime = 0;
+ long newTotalTime = 0;
for (int j = 0; j < runs; j ++) {
long start = System.nanoTime();
@@ -64,6 +70,7 @@ public class TestELParserPerformance extends ELBaseTest {
stack.push(parser);
}
long end = System.nanoTime();
+ reinitTotalTime += (end - start);
System.out.println(parseIterations +
" iterations using ELParser.ReInit(...) took " + (end - start) + "ns");
@@ -76,9 +83,12 @@ public class TestELParserPerformance extends ELBaseTest {
parser.CompositeExpression();
}
long end = System.nanoTime();
+ newTotalTime += (end - start);
System.out.println(parseIterations +
" iterations using new ELParser(...) took " + (end - start) + "ns");
}
+
+ Assert.assertTrue("Using new ElParser() was faster then using ELParser.ReInit", reinitTotalTime < newTotalTime);
}
}
diff --git a/test/org/apache/juli/TestOneLineFormatterPerformance.java b/test/org/apache/juli/TestOneLineFormatterPerformance.java
index 54b1e33ebd..a90ebd6529 100644
--- a/test/org/apache/juli/TestOneLineFormatterPerformance.java
+++ b/test/org/apache/juli/TestOneLineFormatterPerformance.java
@@ -16,6 +16,7 @@
*/
package org.apache.juli;
+import org.junit.Assert;
import org.junit.Test;
/**
@@ -28,19 +29,29 @@ public class TestOneLineFormatterPerformance {
@Test
public void testDateFormat() throws Exception {
+ DateFormat stringFormatImpl = new StringFormatImpl();
+ long stringFormatImplTime = doTestDateFormat(stringFormatImpl);
+
+ DateFormat dateFormatCacheImpl = new DateFormatCacheImpl();
+ long dateFormatCacheImplTime = doTestDateFormat(dateFormatCacheImpl);
+
+ Assert.assertTrue("String#format was faster that DateFormatCache",
+ dateFormatCacheImplTime < stringFormatImplTime);
+ }
+
+
+ private long doTestDateFormat(DateFormat df) {
int iters = 1000000;
- DateFormat[] dfs = new DateFormat[] { new StringFormatImpl(), new DateFormatCacheImpl() };
-
- for (DateFormat df : dfs) {
- long start = System.nanoTime();
- for (int i = 0; i < iters; i++) {
- df.format(System.nanoTime());
- }
- long end = System.nanoTime();
- System.out.println(
- "Impl: [" + df.getClass().getName() + "] took [" + (end-start) + "] ns");
+
+ long start = System.nanoTime();
+ for (int i = 0; i < iters; i++) {
+ df.format(System.nanoTime());
}
+ long end = System.nanoTime();
+ System.out.println(
+ "Impl: [" + df.getClass().getName() + "] took [" + (end - start) + "] ns");
+ return end - start;
}
diff --git a/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java b/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java
index 2ad8f7f142..f17098488e 100644
--- a/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java
+++ b/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java
@@ -21,29 +21,27 @@ import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
+import org.junit.Assert;
import org.junit.Test;
+/*
+ * This is a relative performance test so it remains part of the standard test run.
+ */
public class TestCharsetCachePerformance {
@Test
- public void testNoCsCache() throws Exception {
- doTest(new NoCsCache());
- }
-
-
- @Test
- public void testFullCsCache() throws Exception {
- doTest(new FullCsCache());
+ public void testCache() throws Exception {
+ long timeNone = doTest(new NoCsCache());
+ long timeFull = doTest(new FullCsCache());
+ long timeLazy = doTest(new LazyCsCache());
+
+ Assert.assertTrue("No cache was faster than full cache", timeFull < timeNone);
+ Assert.assertTrue("No cache was faster than lazy cache", timeLazy < timeNone);
+ Assert.assertTrue("Lazy cache was faster than full cache ", timeFull < timeLazy);
}
- @Test
- public void testLazyCsCache() throws Exception {
- doTest(new LazyCsCache());
- }
-
-
- private void doTest(CsCache cache) throws Exception {
+ private long doTest(CsCache cache) throws Exception {
int threadCount = 10;
int iterations = 10000000;
String[] lookupNames = new String[] {
@@ -68,6 +66,8 @@ public class TestCharsetCachePerformance {
long endTime = System.nanoTime();
System.out.println(cache.getClass().getName() + ": " + (endTime - startTime) + "ns");
+
+ return endTime - startTime;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 05/08: Comment this absolute performance test with an upper limit
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit 6be9e259d678f85604c561cf45f8d970d6139075
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:09:29 2024 +0000
Comment this absolute performance test with an upper limit
Since there is an upper limit this test can pass/fail so it is included
in the standard test run.
---
test/org/apache/catalina/mapper/TestMapperPerformance.java | 3 +++
1 file changed, 3 insertions(+)
diff --git a/test/org/apache/catalina/mapper/TestMapperPerformance.java b/test/org/apache/catalina/mapper/TestMapperPerformance.java
index 606794397f..4b4feb8f81 100644
--- a/test/org/apache/catalina/mapper/TestMapperPerformance.java
+++ b/test/org/apache/catalina/mapper/TestMapperPerformance.java
@@ -21,6 +21,9 @@ import org.junit.Test;
import org.apache.tomcat.util.buf.MessageBytes;
+/*
+ * This is an absolute performance test with an upper limit. Therefore it is executed as part of a standard test run.
+ */
public class TestMapperPerformance extends TestMapper {
@Test
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 04/08: Add additional comment to this relative performance test
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit a574b5af78fb54ab934bca32bac7757be230c0a4
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 12:08:41 2024 +0000
Add additional comment to this relative performance test
---
test/org/apache/catalina/connector/TestResponsePerformance.java | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/test/org/apache/catalina/connector/TestResponsePerformance.java b/test/org/apache/catalina/connector/TestResponsePerformance.java
index 100a20b38f..62c2e1db92 100644
--- a/test/org/apache/catalina/connector/TestResponsePerformance.java
+++ b/test/org/apache/catalina/connector/TestResponsePerformance.java
@@ -24,6 +24,10 @@ import org.junit.Test;
import org.apache.catalina.startup.LoggingBaseTest;
import org.apache.tomcat.unittest.TesterRequest;
+/*
+ * This is a relative performance test so it remains part of the standard test run. If the test fails then we need to
+ * looking at why and possibly disable/remove the homebrew approach for some OS/Java combinations.
+ */
public class TestResponsePerformance extends LoggingBaseTest {
private static final int ITERATIONS = 1000000;
@@ -38,8 +42,8 @@ public class TestResponsePerformance extends LoggingBaseTest {
doHomebrew(resp);
doUri();
- // Note: With Java 11 the 'homebrew' approach is consistently 3 to 4
- // times faster on both MacOS and Linux
+ // Note: With Java 11 the 'homebrew' approach is consistently 3-4 times faster on both MacOS (Intel) and Linux
+ // With Java 22 EA the 'homebrew' approach is consistently a little over 2x faster on MacOS (M1)
// To allow for timing differences between runs, a "best of n" approach
// is taken for this test
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
(tomcat) 08/08: Don't test lazy vs full as they can be quite close
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
markt pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tomcat.git
commit d24302cb1218f66fd59192091f8bccda1acf4ebe
Author: Mark Thomas <ma...@apache.org>
AuthorDate: Mon Jan 8 13:46:16 2024 +0000
Don't test lazy vs full as they can be quite close
---
test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java b/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java
index f17098488e..0a0646db39 100644
--- a/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java
+++ b/test/org/apache/tomcat/util/buf/TestCharsetCachePerformance.java
@@ -37,7 +37,8 @@ public class TestCharsetCachePerformance {
Assert.assertTrue("No cache was faster than full cache", timeFull < timeNone);
Assert.assertTrue("No cache was faster than lazy cache", timeLazy < timeNone);
- Assert.assertTrue("Lazy cache was faster than full cache ", timeFull < timeLazy);
+ // On average full cache is faster than lazy cache but they are close enough the test will fail sometimes
+ //Assert.assertTrue("Lazy cache was faster than full cache ", timeFull < timeLazy);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org