You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2024/10/14 13:29:24 UTC
(pulsar) branch branch-3.0 updated (e88474958ba -> 26f4a400c05)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
from e88474958ba [fix][broker] normalize path (#23438)
new cddb3ac1196 [fix][test] Fix flaky GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic (#23259)
new 40f83aa7947 [fix][test] Fix flaky test ManagedLedgerTest.testDeleteCurrentLedgerWhenItIsClosed (#23437)
new 323cf35c8cc [fix][build] Add basic support for vscode-java and Eclipse IDE (#23448)
new de9c06553b8 [fix][client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled (#23452)
new 26cbc25e7a2 [fix][test] Address flaky GetPartitionMetadataMultiBrokerTest (#23456)
new 26f4a400c05 [improve][broker] Add log to track issue when `handleGetTopicsOfNamespace` (#23434)
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitignore | 2 +
.../bookkeeper/mledger/impl/ManagedLedgerTest.java | 5 ++-
pom.xml | 46 ++++++++++++++++++++++
pulsar-broker/pom.xml | 19 +++++++++
.../apache/pulsar/broker/service/ServerCnx.java | 4 +-
.../admin/GetPartitionMetadataMultiBrokerTest.java | 4 +-
.../broker/admin/GetPartitionMetadataTest.java | 9 ++++-
.../apache/pulsar/client/api/ConsumerBuilder.java | 16 +++++---
pulsar-common/pom.xml | 32 +++++++++++++++
pulsar-transaction/coordinator/pom.xml | 34 +++++++++++++++-
10 files changed, 155 insertions(+), 16 deletions(-)
(pulsar) 06/06: [improve][broker] Add log to track issue when `handleGetTopicsOfNamespace` (#23434)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 26f4a400c0582ad5a78a097363729a52197767fa
Author: Jiwei Guo <te...@apache.org>
AuthorDate: Mon Oct 14 19:19:05 2024 +0800
[improve][broker] Add log to track issue when `handleGetTopicsOfNamespace` (#23434)
(cherry picked from commit 33475ec456ad4f297d8c77bd33b76db38bc80cef)
---
.../src/main/java/org/apache/pulsar/broker/service/ServerCnx.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
index b180d549f82..beef7819589 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java
@@ -2390,8 +2390,8 @@ public class ServerCnx extends PulsarHandler implements TransportCnx {
filterTopics = true;
filteredTopics = TopicList.filterTopics(filteredTopics, topicsPattern.get());
} else {
- log.info("[{}] Subscription pattern provided was longer than maximum {}.",
- remoteAddress, maxSubscriptionPatternLength);
+ log.info("[{}] Subscription pattern provided [{}] was longer than maximum {}.",
+ remoteAddress, topicsPattern.get(), maxSubscriptionPatternLength);
}
}
String hash = TopicList.calculateHash(filteredTopics);
(pulsar) 01/06: [fix][test] Fix flaky GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic (#23259)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit cddb3ac11964fc58e69b49b8d1f29e47df2e6338
Author: hanmz <gu...@tencent.com>
AuthorDate: Wed Oct 9 22:46:47 2024 +0800
[fix][test] Fix flaky GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic (#23259)
(cherry picked from commit c891eaa38a02dbbdb855b5d5d1de609f1ceb6132)
---
.../pulsar/broker/admin/GetPartitionMetadataMultiBrokerTest.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataMultiBrokerTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataMultiBrokerTest.java
index 60691203e77..d1eeabdb3d7 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataMultiBrokerTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataMultiBrokerTest.java
@@ -256,8 +256,8 @@ public class GetPartitionMetadataMultiBrokerTest extends GetPartitionMetadataTes
// Initialize the connections of internal Pulsar Client.
PulsarClientImpl client1 = (PulsarClientImpl) pulsar1.getClient();
PulsarClientImpl client2 = (PulsarClientImpl) pulsar2.getClient();
- client1.getLookup(pulsar2.getBrokerServiceUrl()).getBroker(TopicName.get(DEFAULT_NS + "/tp1"));
- client2.getLookup(pulsar1.getBrokerServiceUrl()).getBroker(TopicName.get(DEFAULT_NS + "/tp1"));
+ client1.getLookup(pulsar2.getBrokerServiceUrl()).getBroker(TopicName.get(DEFAULT_NS + "/tp1")).join();
+ client2.getLookup(pulsar1.getBrokerServiceUrl()).getBroker(TopicName.get(DEFAULT_NS + "/tp1")).join();
// Inject a not support flag into the connections initialized.
Field field = ClientCnx.class.getDeclaredField("supportsGetPartitionedMetadataWithoutAutoCreation");
(pulsar) 03/06: [fix][build] Add basic support for vscode-java and Eclipse IDE (#23448)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 323cf35c8ccfdee84f0a82c0f736b3d2da74e8e9
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Oct 14 12:03:57 2024 +0300
[fix][build] Add basic support for vscode-java and Eclipse IDE (#23448)
(cherry picked from commit 209fd784765916e28d38e50e52f6ed90d47527fd)
---
.gitignore | 2 ++
pom.xml | 46 ++++++++++++++++++++++++++++++++++
pulsar-broker/pom.xml | 19 ++++++++++++++
pulsar-common/pom.xml | 32 +++++++++++++++++++++++
pulsar-transaction/coordinator/pom.xml | 34 +++++++++++++++++++++++--
5 files changed, 131 insertions(+), 2 deletions(-)
diff --git a/.gitignore b/.gitignore
index 80d760cd29d..fe6e44915e6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -99,3 +99,5 @@ test-reports/
.mvn/.gradle-enterprise/
# Gradle Develocity
.mvn/.develocity/
+.vscode
+effective-pom.xml
diff --git a/pom.xml b/pom.xml
index e0fe058048f..eb9e6ff05cc 100644
--- a/pom.xml
+++ b/pom.xml
@@ -308,6 +308,7 @@ flexible messaging model and an intuitive client API.</description>
<errorprone-slf4j.version>0.1.4</errorprone-slf4j.version>
<j2objc-annotations.version>1.3</j2objc-annotations.version>
<lightproto-maven-plugin.version>0.4</lightproto-maven-plugin.version>
+ <build-helper-maven-plugin.version>3.6.0</build-helper-maven-plugin.version>
<dependency-check-maven.version>10.0.2</dependency-check-maven.version>
<roaringbitmap.version>1.2.0</roaringbitmap.version>
<extra-enforcer-rules.version>1.6.1</extra-enforcer-rules.version>
@@ -1807,6 +1808,7 @@ flexible messaging model and an intuitive client API.</description>
<exclude>generated-site/**</exclude>
<exclude>**/*.md</exclude>
<exclude>**/.idea/**</exclude>
+ <exclude>**/.vscode/**</exclude>
<exclude>**/.mvn/**</exclude>
<exclude>**/generated/**</exclude>
<exclude>**/zk-3.5-test-data/*</exclude>
@@ -1951,6 +1953,7 @@ flexible messaging model and an intuitive client API.</description>
<exclude>**/SecurityAuth.audit*</exclude>
<exclude>**/site2/**</exclude>
<exclude>**/.idea/**</exclude>
+ <exclude>**/.vscode/**</exclude>
<exclude>**/.mvn/**</exclude>
<exclude>**/*.a</exclude>
<exclude>**/*.so</exclude>
@@ -2154,6 +2157,49 @@ flexible messaging model and an intuitive client API.</description>
<!-- <nvdDatafeedUrl>https://jeremylong.github.io/DependencyCheck/hb_nvd/</nvdDatafeedUrl> -->
</configuration>
</plugin>
+ <!--
+ vscode-java and Eclipse Maven integration workaround:
+ - Addresses maven-dependency-plugin exception (MDEP-187) issue by ignoring the plugin execution
+ - Uses org.eclipse.m2e:lifecycle-mapping:1.0.0 as configuration placeholder in pluginManagement section
+ For details: https://eclipse.dev/m2e/documentation/m2e-execution-not-covered.html#ignore-plugin-goal
+ -->
+ <plugin>
+ <groupId>org.eclipse.m2e</groupId>
+ <artifactId>lifecycle-mapping</artifactId>
+ <version>1.0.0</version>
+ <configuration>
+ <lifecycleMappingMetadata>
+ <pluginExecutions>
+ <pluginExecution>
+ <pluginExecutionFilter>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <versionRange>[1.0.0,)</versionRange>
+ <goals>
+ <goal>copy</goal>
+ </goals>
+ </pluginExecutionFilter>
+ <action>
+ <ignore />
+ </action>
+ </pluginExecution>
+ <pluginExecution>
+ <pluginExecutionFilter>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <versionRange>[1.0.0,)</versionRange>
+ <goals>
+ <goal>unpack</goal>
+ </goals>
+ </pluginExecutionFilter>
+ <action>
+ <ignore />
+ </action>
+ </pluginExecution>
+ </pluginExecutions>
+ </lifecycleMappingMetadata>
+ </configuration>
+ </plugin>
</plugins>
</pluginManagement>
<extensions>
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index 7953c871763..0dd50dd97bf 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -657,6 +657,25 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <id>add-source</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>target/generated-sources/lightproto/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
diff --git a/pulsar-common/pom.xml b/pulsar-common/pom.xml
index 258aa718b47..3f40f1fb2e5 100644
--- a/pulsar-common/pom.xml
+++ b/pulsar-common/pom.xml
@@ -293,6 +293,38 @@
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <id>add-source</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>target/generated-sources/protobuf/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ <execution>
+ <id>add-test-source</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-test-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>target/generated-test-sources/protobuf/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+
<plugin>
<groupId>pl.project13.maven</groupId>
diff --git a/pulsar-transaction/coordinator/pom.xml b/pulsar-transaction/coordinator/pom.xml
index 73764ed7b17..923e52774a3 100644
--- a/pulsar-transaction/coordinator/pom.xml
+++ b/pulsar-transaction/coordinator/pom.xml
@@ -61,7 +61,7 @@
</dependency>
</dependencies>
-
+
<build>
<plugins>
<plugin>
@@ -117,7 +117,37 @@
</execution>
</executions>
</plugin>
-
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>build-helper-maven-plugin</artifactId>
+ <version>${build-helper-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <id>add-source</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>target/generated-sources/protobuf/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ <execution>
+ <id>add-test-source</id>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>add-test-source</goal>
+ </goals>
+ <configuration>
+ <sources>
+ <source>target/generated-test-sources/protobuf/java</source>
+ </sources>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
<plugin>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-maven-plugin</artifactId>
(pulsar) 05/06: [fix][test] Address flaky GetPartitionMetadataMultiBrokerTest (#23456)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 26cbc25e7a244a3c6a512f5cfc1c4f99fa3e2997
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Oct 14 14:09:25 2024 +0300
[fix][test] Address flaky GetPartitionMetadataMultiBrokerTest (#23456)
(cherry picked from commit 0c4108b8208bc84761ca8d03bd111b46ca39bdf8)
---
.../org/apache/pulsar/broker/admin/GetPartitionMetadataTest.java | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataTest.java
index e9a639697d9..d06f6ea1c56 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataTest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/GetPartitionMetadataTest.java
@@ -47,6 +47,7 @@ import org.apache.pulsar.common.policies.data.ClusterDataImpl;
import org.apache.pulsar.common.policies.data.TenantInfoImpl;
import org.apache.pulsar.common.policies.data.TopicType;
import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.tests.TestRetrySupport;
import org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble;
import org.awaitility.Awaitility;
import org.testng.annotations.AfterClass;
@@ -56,7 +57,7 @@ import org.testng.annotations.Test;
@Test(groups = "broker-admin")
@Slf4j
-public class GetPartitionMetadataTest {
+public class GetPartitionMetadataTest extends TestRetrySupport {
protected static final String DEFAULT_NS = "public/default";
@@ -72,8 +73,10 @@ public class GetPartitionMetadataTest {
protected PulsarClientImpl clientWithHttpLookup1;
protected PulsarClientImpl clientWitBinaryLookup1;
+ @Override
@BeforeClass(alwaysRun = true)
protected void setup() throws Exception {
+ incrementSetupNumber();
bkEnsemble = new LocalBookkeeperEnsemble(3, 0, () -> 0);
bkEnsemble.start();
// Start broker.
@@ -85,8 +88,10 @@ public class GetPartitionMetadataTest {
admin1.namespaces().createNamespace(DEFAULT_NS);
}
+ @Override
@AfterClass(alwaysRun = true)
protected void cleanup() throws Exception {
+ markCurrentSetupNumberCleaned();
cleanupBrokers();
if (bkEnsemble != null) {
bkEnsemble.stop();
@@ -236,7 +241,7 @@ public class GetPartitionMetadataTest {
String pulsarUrl = pulsar1.getBrokerServiceUrl();
PulsarClientImpl[] clients = getClientsToTest(false);
for (PulsarClientImpl client : clients) {
- client.getLookup(pulsarUrl).getBroker(TopicName.get(DEFAULT_NS + "/tp1"));
+ client.getLookup(pulsarUrl).getBroker(TopicName.get(DEFAULT_NS + "/tp1")).join();
}
// Inject a not support flag into the connections initialized.
Field field = ClientCnx.class.getDeclaredField("supportsGetPartitionedMetadataWithoutAutoCreation");
(pulsar) 02/06: [fix][test] Fix flaky test ManagedLedgerTest.testDeleteCurrentLedgerWhenItIsClosed (#23437)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 40f83aa794708720ba6a12eb8b613481f71978de
Author: Jiawen Wang <74...@users.noreply.github.com>
AuthorDate: Sun Oct 13 22:56:55 2024 +0800
[fix][test] Fix flaky test ManagedLedgerTest.testDeleteCurrentLedgerWhenItIsClosed (#23437)
(cherry picked from commit aa6bc0929bf28f5e8498ddb2055057b078d46481)
---
.../java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
index 7b648c11e33..de51f63e06c 100644
--- a/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
+++ b/managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java
@@ -4358,9 +4358,10 @@ public class ManagedLedgerTest extends MockedBookKeeperTestCase {
// Detect the current ledger is full by the timed task. (Imitate: the timed task `checkLedgerRollTask` call
// `rollCurrentLedgerIfFull` periodically).
ml.rollCurrentLedgerIfFull();
- // the ledger closing in the `rollCurrentLedgerIfFull` is async, so the wait is needed.
- Awaitility.await().untilAsserted(() -> assertEquals(ml.ledgers.size(), 2));
}
+ // wait the new ledger create
+ Awaitility.await().untilAsserted(() -> assertEquals(ml.ledgers.size(), 2));
+
// Act: Trigger trimming to delete the previous current ledger.
ml.internalTrimLedgers(false, Futures.NULL_PROMISE);
// Verify: A new ledger will be opened after the current ledger is closed and the previous current ledger can be
(pulsar) 04/06: [fix][client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled (#23452)
Posted by lh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit de9c06553b8ff600c6b67af5e1bebf3b8d8b03a5
Author: Lari Hotari <lh...@users.noreply.github.com>
AuthorDate: Mon Oct 14 13:19:19 2024 +0300
[fix][client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled (#23452)
(cherry picked from commit e2fb0daac8411d76a3cca8df3e8f9f2b5c9b4ed6)
---
.../org/apache/pulsar/client/api/ConsumerBuilder.java | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java
index ff1a3cd73bb..eda5b1c41a9 100644
--- a/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java
+++ b/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java
@@ -200,17 +200,21 @@ public interface ConsumerBuilder<T> extends Cloneable {
ConsumerBuilder<T> ackTimeout(long ackTimeout, TimeUnit timeUnit);
/**
- * Acknowledgement returns receipt, but the message is not re-sent after getting receipt.
+ * Enables or disables the acknowledgment receipt feature.
*
- * Configure the acknowledgement timeout mechanism to redeliver the message if it is not acknowledged after
- * ackTimeout, or to execute a timer task to check the acknowledgement timeout messages during every
- * ackTimeoutTickTime period.
+ * <p>When this feature is enabled, the consumer ensures that acknowledgments are processed by the broker by
+ * waiting for a receipt from the broker. Even when the broker returns a receipt, it doesn't guarantee that the
+ * message won't be redelivered later due to certain implementation details.
+ * It is recommended to use the asynchronous {@link Consumer#acknowledgeAsync(Message)} method for acknowledgment
+ * when this feature is enabled. This is because using the synchronous {@link Consumer#acknowledge(Message)} method
+ * with acknowledgment receipt can cause performance issues due to the round trip to the server, which prevents
+ * pipelining (having multiple messages in-flight). With the asynchronous method, the consumer can continue
+ * consuming other messages while waiting for the acknowledgment receipts.
*
- * @param isAckReceiptEnabled {@link Boolean} enables acknowledgement for receipt
+ * @param isAckReceiptEnabled {@code true} to enable acknowledgment receipt, {@code false} to disable it
* @return the consumer builder instance
*/
ConsumerBuilder<T> isAckReceiptEnabled(boolean isAckReceiptEnabled);
-
/**
* Define the granularity of the ack-timeout redelivery.
*