You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/23 14:34:37 UTC
[GitHub] [pulsar] eolivelli opened a new pull request #9043: set testRetryCount default to 0
eolivelli opened a new pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043
Motivation;:
- testRetryCount system property is used by a custom IRetryAnalyzer
- unfortunately in case of retry the Before/After hooks are not executed and the system is in an unpredictable status during retries
- it is better to change the default to 0 and let the user or CI configuration to opt-in explicitly this feature
- Maven Surefire plugin has a specific standard configuration option, it is better to use that option
Changes:
set the default value to 0, both in pom.xml and in Java code
If you want to use this feature you can simply run your build with -DtestRetryCount=1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#discussion_r553783287
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
@sijie if we use `BeforeMethod` the behaviour is correct, I mean that the framework correctly disposes the env and then it set it up correctly for the second trial.
Unfortunately many of our tests use BeforeClass/AfterClass, that makes sense because we do not want to start a PulsarService for each test.
In my opinion it is better to leave this testRetryCount to 0, and let the developer choose locally to set it to an higher value.
If we want to keep testRetryCount to 1 we should fix all of the tests and switch from BeforeClass to BeforeMethod, but this change will have the effect of making the suite use more resources and take more time to complete.
If we have flaky tests (that are not so many) we will work to fix them, I can help here.
Currently I don't like very much the status of our suite, especially **because when a test fails the failure is hidden by an unrelated failure of the second trial**
I want to help to make it work better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-754639923
This is an example on a PR validation.
The test failed, but on GitHub we see a failure about the 'second trial', and the failure is unrelated to the test itself but to the environment.
The failed trial is reported as "Skip"

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a change in pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#discussion_r548439970
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
@sijie this test explains my point.
when TestNG sees the first failure it executes again **only the test method**
during the second execution a subscription with name "mysub" is already registered.
This behaviour makes it useless to rerun a flaky test, because it is possible that the second time the test passes not because the failure was due to some bad design of the test, but because the polluted env enables the test to pass.
In my opinion this is very dangerous and make the suite less reliable overall.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#discussion_r553807127
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
If `testRetryCount == 1` is a problem, you can find those tests impacted by this and fix those tests.
I am just not convinced by `testRetryCount == 0` will help us get into a better situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-751449338
@sijie @merlimat is my example useful in order to understand the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-750470888
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#discussion_r553806131
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
If we use `BeforeClass`, why not take a different approach to fix the tests to make it be able to run multiple tests? In this way, you can also help improve the project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-754531190
@codelipenghui that do you think about this patch ?
@sijie is my point convincing ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-750327559
@merlimat @ivankelly it looks like you worked in these file recently, please take a look
@lhotari I think that this fix will make the tests more reliable, probably uncovering new flaky tests, but rerunning a test in an unknown environment is far worse that seeing a flaky test.
Flaky tests can be a symptom that something is going wrong, we should not rely on this kind of tricks
cc @sijie @rdhabalia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli closed pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli closed pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#discussion_r553740443
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
You are using BeforeClass not BeforeMethod. `BeforeClass` only executes when the test was initialized.
Have you tried `BeforeMethod`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-750463536
I can set the rerunFailureTests option in surefire.
The way this option is implemented is not reliable.
Otherwise I can add the option to the command line of every github actions build
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#issuecomment-750469815
@eolivelli If we are going to set the option for every GitHub action, why do we want to change the default value? For people who want to disable retryCount, they can just set the retryCount to `0` themselves. In other words, I don't know what exactly the value this PR brings. If you can show it improves the stability of CI, I am fine with it. Otherwise, I would hold on this change, because people can disable the retryCount themselves.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #9043: set testRetryCount default to 0
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #9043:
URL: https://github.com/apache/pulsar/pull/9043#discussion_r553806131
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
If we use `BeforeClass`, why not take a different approach to fix the tests to make it be able to run multiple tests? In this way, you can also help improve the project.
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/DemoPollutedEnvTest.java
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.service;
+
+import org.apache.pulsar.client.api.Consumer;
+import org.apache.pulsar.client.api.Schema;
+import static org.testng.Assert.fail;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+/**
+ * This demo shows that the test is executed twice, but the environment
+ * in the second execution is not a fresh new env,
+ * in fact the consumer is not able to subscribe.
+ *
+ * org.apache.pulsar.client.api.PulsarClientException$ConsumerBusyException: Exclusive consumer is already connected
Review comment:
If `testRetryCount == 1` is a problem, you can find those tests impacted by this and fix those tests.
I am just not convinced by `testRetryCount == 0` will help us get into a better situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org