You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Semb Wever (Jira)" <ji...@apache.org> on 2021/04/11 10:38:00 UTC
[jira] [Comment Edited] (CASSANDRA-16587) flaky
AntiCompactionBytemanTest.testRedundantTransitions byteman Connection
refused
[ https://issues.apache.org/jira/browse/CASSANDRA-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318726#comment-17318726 ]
Michael Semb Wever edited comment on CASSANDRA-16587 at 4/11/21, 10:37 AM:
---------------------------------------------------------------------------
I can see two different paths forward here.
1) Admit default that our different 'test' targets are not parallel JVM friendly, and remove everything related to 'testparallel' and 'test.runners' from the build.xml. This would solve a number of problems: distributed in-jvm tests, conf files, and byteman agent.
2) Fix the issues one-by-one, the best we can…
For (1), the performance issue is of concern: running tests one at a time on ci-cassandra agents that have 4 cores and 16gb ram available to each executor is a waste of resources and time. Going from one test.runner to two on these agents, doubled the performance. If we were to remove test parallelism from the build.xml (like [this patch|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587-2/trunk]) it could be replaced with parallelism at the docker level, example patch [here|https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16587-2/trunk].
For (2) the following patch, which is 100% completely reliable but shows a lot of promise (see CI), makes byteman tests wait until the byteman port becomes available before running the test…
- https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587/trunk
CI [!https://ci-cassandra.apache.org/job/Cassandra-devbranch/616/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/616/pipeline]
was (Author: michaelsembwever):
I can see two different paths forward here.
1) Admit default that our different 'test' targets are not parallel JVM friendly, and remove everything related to 'testparallel' and 'test.runners' from the build.xml. This would solve a number of problems: distributed in-jvm tests, conf files, and byteman agent.
2) Fix the issues on-by-one, the best we can…
For (1), the performance issue is of concern: running tests one at a time on ci-cassandra agents that have 4 cores and 16gb ram available to each executor is a waste of resources and time. Going from one test.runner to two on these agents, doubled the performance. If we were to remove test parallelism from the build.xml (like [this patch|https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587-2/trunk]) it could be replaced with parallelism at the docker level, example patch [here|https://github.com/apache/cassandra-builds/compare/trunk...thelastpickle:mck/16587-2/trunk].
For (2) the following patch, which is 100% completely reliable but shows a lot of promise (see CI), makes byteman tests wait until the byteman port becomes available before running the test…
- https://github.com/apache/cassandra/compare/trunk...thelastpickle:mck/16587/trunk
CI [!https://ci-cassandra.apache.org/job/Cassandra-devbranch/616/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/616/pipeline]
> flaky AntiCompactionBytemanTest.testRedundantTransitions byteman Connection refused
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-16587
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16587
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
> Reporter: Michael Semb Wever
> Priority: Normal
> Fix For: 4.0.x
>
>
> Failing occasionally with
> {noformat}
> java.net.ConnectException: Connection refused (Connection refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:607)
> at java.net.Socket.connect(Socket.java:556)
> at java.net.Socket.<init>(Socket.java:452)
> at java.net.Socket.<init>(Socket.java:229)
> at org.jboss.byteman.agent.submit.Submit$Comm.<init>(Submit.java:881)
> at org.jboss.byteman.agent.submit.Submit.submitRequest(Submit.java:787)
> at org.jboss.byteman.agent.submit.Submit.addScripts(Submit.java:603)
> at org.jboss.byteman.contrib.bmunit.BMUnit.loadScriptText(BMUnit.java:291)
> at org.jboss.byteman.contrib.bmunit.BMUnitRunner$9.evaluate(BMUnitRunner.java:362)
> at org.jboss.byteman.contrib.bmunit.BMUnitRunner$6.evaluate(BMUnitRunner.java:263)
> at org.jboss.byteman.contrib.bmunit.BMUnitRunner$1.evaluate(BMUnitRunner.java:97)
> {noformat}
> Looking through the [logs |https://nightlies.apache.org/cassandra/trunk/Cassandra-trunk-test-cdc/650/Cassandra-trunk-test-cdc/jdk=jdk_11_latest,label=cassandra,split=6/build/test/logs/cdc/TEST-org.apache.cassandra.db.compaction.AntiCompactionBytemanTest.log.xz] this correlates to
> {noformat}
> Setting org.jboss.byteman.allow.config.update=true
> TransformListener() : unexpected exception opening server socket java.net.BindException: Address already in use (Bind failed)
> java.net.BindException: Address already in use (Bind failed)
> at java.net.PlainSocketImpl.socketBind(Native Method)
> at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
> at java.net.ServerSocket.bind(ServerSocket.java:390)
> at java.net.ServerSocket.bind(ServerSocket.java:344)
> at org.jboss.byteman.agent.TransformListener.initialize(TransformListener.java:81)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.jboss.byteman.agent.Main.premain(Main.java:286)
> at org.jboss.byteman.agent.Main.agentmain(Main.java:309)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
> at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org